When Product Ratings Are a Problem

The worst summer in 20 years, movie box office revenue from May to Labor Day in 2017 was 15% less than the same dates last year. The reason could be lower film quality. Or it could be Netflix and Amazon.

But the film studios blame Rotten Tomatoes.

They say that the Tomatometer can make or break a film. A movie just needs to get a “rotten” rating of below 60 and millions of moviegoers will avoid it.

So, when King Arthur: Legend of the Sword got a 27, the movie studio was not pleased:

Where are we going? To ratings inflation.

High and Low Ratings

Movies

To check out a film, we can go to Rotten Tomatoes, Metacritic, IMDb or Fandango. In 2015, the statistics website fivethirtyeight suggested we stick with the first three “review aggregators” because they tend to use more of their ratings scale, are pretty consistent and avoid outside interference.

On the other hand, they indicated that Fandango was less dependable. You can see below that Fandango’s reviews lay primarily in 5-star territory while ratings from Rotten Tomatoes and Metacritic were far lower:

From fivethirtyeight.com

Uber

In a 2016 Time article, a journalist said he regretted giving his driver a “3” rating when he learned it was a “bad” grade. To stay with Uber, drivers need to maintain a 4.6 out of 5. Knowing that, many of us might limit the range of our reviews to that 4.6-5 so as not to harm the driver.

In its newsletter, Uber reminds drivers that below 4.6 is underperformance:

Similarly, researchers from Boston University concluded that 95% of 600,000 Airbnb reviews were 4.5 stars or more out of 5. And at Amazon, for 2014, consumers gave 5 stars in more than half of 1.2 million reviews.

You can see that a 5-star rating dominates at Airbnb:

Our Bottom Line:  Inflation

Like prices and grades, ratings can be called inflationary when they distort value.

With prices, we could have too many dollars trying to pay for too few goods and services. The result is a higher price for the same item. If prices rise considerably, they no longer tell us anything about a product.

For grades, inflation occurs when everyone gets an A or a B. High grades for all eliminate the message a grade can convey. They no longer distinguish between high and low quality work.

Similarly, film, Uber and Airbnb ratings need to be high and low to reflect quality.

And for that reason, we should thank Rotten Tomatoes.

My sources and more: Thanks to the NY Times for their movie ratings article. But for the real analysis I suggest this Boston University study. Then, if you would like to read more about ratings, do look at Businessinsider, fivethirtyeight and Time.

Please note that several sentences in this post were in a past econlife.