Why Metrics Can Be Misleading

In the former Soviet Union, factory quotas determined output in ways we might not expect. When the metric for a lamp factory  was a certain number of kilograms a year, each lamp was unusually heavy. Satirizing a weight quota, this cartoon shows us a giant nail. I have been told that the caption says, “The month’s plan fulfilled:”

To see the ways that metrics mislead, we can leave the former Soviet Union and look closer to home.

Metric Surprises

Discussing misleading metrics, economist Tim Harford tells us, “We hit the target but miss the point.”

In 2005 or so, British Prime Minister Tony Blair proclaimed a 48-hour rule that assured National Health Service patients they could get an appointment within two days when they called a physician. Responding, physicians refused to book advance appointments. Instead, they wanted to retain open slots for booking people within 48 hours. Unable to call in advance, patients wound up with long phone waits on the days they wanted an exam.

Somewhat similarly, New Yorker writer and physician Atul Gawande tells us the unintended consequences of the Apgar score. A metric that assesses the well-being of a newborn, the score measures color, breathing, and other key signs of health. However, hospitals soon discovered their Apgar averages could rise if babies were delivered through C-sections rather than using forceps. The first requires slicing open the mom and removing a more “pristine” infant; the second is natural and can be messy. As you might imagine, the C-section produces more trauma and a longer recovery for the mother.

We also can look at school ranking or snakes.

Because colleges care about their U.S.News rank, many focus on the variables that bring them closer to the top, even when their educational merit is questionable. Similarly, a British bounty on dead cobras (in India) that sought to reduce the snake population wound up increasing it. Resourceful individuals started raising snakes and then killing them.

Our Bottom Line: Incentives

Asked what propels the market system, we could say incentives. On the demand side of transactions, consumers are willing and able to buy more when prices are low. Correspondingly, on the supply side, producers up output when prices rise. For that reason, some say price could be the least misleading metric. As an incentive, it can result in better quality and desirable quantities. But as we know, the market’s prices can have their glitches. So, we can conclude by asking which metrics optimize how an economy produces and distributes its goods and services?

Offering some wisdom about metrics, economist Charles Goodhart (Goodhart’s Law) told us that, “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” He just meant that a measure that becomes a target is no longer a good measure. As Tim Harford warned, you can hit the target but miss the point.

…Like when you wind up with a giant nail.

My sources and more: Thanks to the Hustle for returning me to the misleading metrics debate. From there, economist Tim Harford’s Messy, discussed in this econtalk podcast, has the stories. Finally, while Knowledge at Wharton had the facts, the best article was at Bloomberg.