Why It’s Tough to Evaluate Job Performance

After being told that the number of accidental deaths had to decrease, certain Chinese bureaucrats appear to have quickly improved safety.

Chinese Death Ceilings

In 2004, the State Council of China proclaimed that safety would equal levels in middle income countries by 2020. Using the number of accidental fatalities as one yardstick, SAWS, the State Administration of Work Safety, implemented a program to reduce accidental deaths by 2.5 percent annually with the People’s Daily reporting the data. An incentive for certain local officials was NSNP–No Safety No Promotion.

If more safety will get you a promotion, it makes sense that the 20 provinces with officials who were told NSNP became safer places. Yes?

Not necessarily.

Where are we going? To the unintended consequences of an incentive.

The Impact of No Safety No Promotion

Using statistical anomalies, researchers from USC and Columbia University concluded that China’s saftey numbers might have improved because they were manipulated. Too many deaths, for example, occurred on the eighth day after an accident–a bit fishy when a death is counted as natural only after day seven, The numbers could also look better by shifting their classification, like saying the fatality was caused by disease rather than coal mining.

Looking at this graph, we can ask if China’s accidental deaths plummet because of improved safety or statistical manipulation.

From:”The Economics of Death Ceilings”

Our Bottom Line: Unintended Consequences

The Chinese death study is about a lot more than China. Instead, it probably confirm’s Goodhart’s Law that “A performance metric is only useful as a performance metric as long as it is not used as a performance metric.” The reason? Distorting the activity that it is measuring, the target ceases to be valid.

The implications are monumental. At schools, universal testing is supposed to create the incentive to improve education but some teachers then teach for the test and little else. In cities trying for lower crime rates, law enforcement could feel the pressure to reclassify crime statistics.  And of course, you and I can talk anecdotally about the impact of performance metrics on our job.

Whether looking at safety in China or human capital at work, evaluation targets have not necessarily improved performance even when the numbers reflect success.