Malcolm Gladwell’s Valentine’s Day article for the New Yorker on the importance of criteria weighting when evaluating colleges, football, and cars was more profound and unnoticed by the blogosphere than it had any right to be. It went completely under the radar. We, as bloggers, typically do a stand-up job of pulling in all sorts of events, affairs, and going-ons into the automotive realm, even if, like Jalopnik, they are at best connected to the auto industry by their possession of a motor. But when Gladwell gave a Stone Cold Stunner to one of the icons of print motoring journalism, he shed light on a piece of human psychology that we as petrolheads use as a crutch rather than a platform.

Gladwell is not a demonstratable car enthusiast in any sense. He is primarily a social psychologist and a popularizer of scientific research. His focuses are business, psychology, and the elucidation of surprising patterns in everyday normativeness. Yet, it often takes the perspective of an outsider (not Outlier) to see the decay within.

Lest we should think otherwise, the small group of us who live and breathe cars are an organization, and no organization is free from its own internal struggles. But it is the struggles that aren’t fought that are the most troubling. TTAC dukes it out with Jalopnik for the slightly obsessed among us. They tease one another back and forth over their focus, or even more nit-pickingly, their site designs; but none of this bickering matters one iota when they each occupy a different extreme on the bell curve. TTAC appeals to those who enjoy quality writing, numbers, industry politics, and international news. Jalopnik offers its readers a dose of celebrity culture, car crashes, explosions, and the weird and wonderful creations that man occasionally creates. They really don’t compete at all – in fact, they complement each other rather nicely. Then there’s Autoblog, which takes the road right up the middle and grabs the lion’s share of people who see their car as something even vaguely more than a grocery-getter.

But beneath all of the blogging, the debating, the griping, and the vitriol is the accepted premise cars can and should be rated on a scale. This makes comparisons easy, quantifiable, and digestible at a glance. But what are we rating and how are we doing it?

When CarEnvy.ca was first founded in the bleakly damp depths of the 2008 recession, we reviewed a couple of Lexuses and we broke down the reviews into a number of categories and rated them on each. It was pretty standard fare. These weren’t comparison tests, just reviews in isolation, but breaking down the components of a car review into a set of numbers implied that we had a context with which to base those judgment on. Quite frankly, we didn’t, but we did have an expectation created by brand marketing, other reviews we’d read, and people we’d talked to. These are all very intangible and very powerful persuaders of opinion that lead to some kind of quantification of quality. But what does a transmission that is 6/10 feel like? Is it not as smooth as we’d like, not as fast, not as crisp, not as gentle? What does a 10/10 transmission feel like and how does it accomplish apparent perfection?

These questions can be applied to individual reviews, but they are really most powerful in comparison tests, where writers must justify their chosen victor and vanquish its defeated opponents on the appearance of solid, logical grounds. What Gladwell revealed in his analysis was that we spend much of our time debating the subjective scores used in Car and Driver tests, such as “fun factor”, but that we overlook the categorical weighting that is arguably more important. Does a sports car buyer really base 25% of their decision on practicality, or a truck buyer 20% on fuel economy?

Using a frozen weighting system may seem consistent and reliable, and therefore lend some impression of credibility and trustworthiness, but that doesn’t mean it’s correct.

Take the May 2011 issue for example. As we flicked through the brilliantly composed pages of the C&D iPad app (which offers free issues, by the way), we couldn’t help but notice Gladwell’s point among the 5 comparison tests featured in the issue. The exact same rating system used to compare the Focus/Elantra/Cruze/Mazda3/Jetta in Comparison #1 was used to compare the 1M Coupe/TT-RS/G37 IPL in Comparison #4. Both comparison tests allotted 100 marks for Vehicle, 55 for Powertrain, and 60 for Chassis. Does C&D truly mean to convince its readers that the prospective buyer of a $25k primary vehicle evaluates a car’s attributes the same way one does if they’re were buying a $55k third vehicle? The Chassis category should be nowhere near as important for economy cars and should be much more valuable for coupes with sporting intentions, one would think.

It’s not enough to simply assign marks when comparing two products, it’s clear that the way the marks are assigned makes an equally significant contribution. The current rating systems used by magazines like Car&Driver are ripe for a revolution, one that adapts to the intended reader. As I write this from Bilbao, “viva la revolución” has never been more apt.