← Blog

Why star ratings mislead, and what to measure

data science star ratings

Star ratings were created to show how much we like a product or service. They do a rather poor job of it. In this post, I look at why the five-star system misleads, and why what customers do tells you more than what they rate.

If a product gets a lot of five-star reviews, it must be good, or so the story goes. Let’s look at a few problems with these little golden constellations.

Explicit and implicit feedback

Star ratings are explicit feedback. A user decides to rate a product or service as good or bad. Implicit feedback comes from behavior: watching only the first five minutes of a movie, or watching an entire season in one weekend.

With explicit feedback, the user is in control. That’s part of the problem.

The bias in explicit ratings

When we rate products, we bring a lot of bias. Take the hotels in Tulum, Mexico, in 2019. Seaweed piled up in front of many beachfront resorts. Guests rated those hotels lower, because in some cases the beach was unusable and the smell was bad. That wasn’t the hotels’ fault. And the hotels a block back from the beach, serving the same kind of guests and advertising the same beaches, didn’t take the hit.

Thick bands of sargassum seaweed covering a Caribbean beach while a tractor clears it

So the seaweed caused a big ratings dip for the beachfront hotels through no fault of their own. Their ratings stopped describing the service, the restaurants or the cleanliness.

The same thing happens with one-star reviews for products that arrived a day late or went to the neighbor’s house. Those reviews say nothing about the product. They’re about delivery. And as some merchants remind me on the little paper slips they send with their products, “Your reviews keep our business alive.”

The scale has no room at the top

There’s another issue. The five-star system can’t separate an extraordinary product from one that simply had no flaws. We start by assuming everything deserves five stars. If I rate a ride-share driver three stars or below, I won’t be matched with that driver again, and even four stars counts as bad. So how do I tell the app that a ride was incredible, when the last one simply gave me no reason to complain? I can’t.

What to measure

As shoppers, we end up reading paragraphs of comments to find out why people liked or disliked that beachfront hotel or that new vacuum cleaner. And some of those reviews are paid for.

Implicit feedback is a much better way to learn what customers like, as long as you can measure it. Movies are a perfect example. Netflix began replacing its five-star ratings with a thumbs up and thumbs down in 2017, and it learns far more from how much and how often we watch.

Clothing stores have no such signal. They sell a shirt and never learn how often it gets worn. So they rely on reviews, with all their flaws. Some companies are experimenting with sensors in garments, but the question of how much a customer likes a piece of clothing matters too much to manufacturers to go away soon.

Behavior is what good recommendations are built on. I go deeper in a guide to recommendation systems for business and in how popularity can kill personalization.

It’s worth paying attention to how ratings are used and what they really say about satisfaction. Often, they cover up a more complicated story. If you want help working out which customer signals are worth measuring, the first call with one of our senior engineers is free.

Let's talk about what you want to improve

Talk to an engineer