Ranking products by rating: use the Wilson score

Here’s a common situation that surprisingly few companies get right. Your customers rate products with a thumbs up or a thumbs down, and you want to show the best-liked products first. When you get the order wrong, customers notice, and they trust your ratings less.
The fix is quick, and it’s been my go-to since I first wrote about it in 2019.
What’s this product’s rating?
Say a product has 100 ratings: 80 thumbs up and 20 thumbs down. Is its rating 80%? Do 80% of your customers like it?
The honest answer is that we don’t know, because we haven’t asked everyone. The 80% tells us something, but it’s missing one important piece: how confident we can be in it.
Why confidence matters
Now say you have two products to rank:
- Product A has 80 thumbs up out of 100.
- Product B has two thumbs up out of two.
Product B was liked 100% of the time and product A only 80% of the time. If you sort by the average, B goes first. But two ratings tell you very little. It’s like flipping a coin twice and getting heads both times. That doesn’t mean you have a two-headed coin.
Could product A really deserve a 40% rating? It’s possible, but very unlikely after 100 ratings. Could product B? Easily. So the safer approach is to rank each product by the lowest rating it could reasonably have, given how many ratings it has.
Rank by the lower bound of the Wilson score
The method has a long name: the lower bound of the Wilson score confidence interval. In plain terms, you give it the count of thumbs up, the count of all ratings, and how confident you want to be. It tells you the lowest rating the product is likely to have.
With 95% confidence, here’s what you get:
- Product B (two of two): somewhere between 34.2% and 100%. Its lower bound is 34.2%.
- Product A (80 of 100): somewhere between 71.1% and 86.7%. Its lower bound is 71.1%.
Ranked by the lower bound, product A comes first, which is what most people would expect. Product B moves up as it collects more good ratings.
Your engineers can calculate this in a few lines of Python:
python
from statsmodels.stats.proportion import proportion_confint
print(proportion_confint(2, 2, alpha=0.05, method="wilson")) # (0.342, 1.0)
print(proportion_confint(80, 100, alpha=0.05, method="wilson")) # (0.711, 0.867)
You don’t need to show these values to customers. Use them behind the scenes to order the list. I first learned this from Evan Miller’s How Not To Sort By Average Rating, which also shows how to do it in Excel and SQL.
Where else this helps
The same idea works anywhere a small sample can look better than a large one: reviews, click rates on new products, or response rates on a new email. Star ratings bring their own problems, which I cover in the trouble with star ratings.
Ranking also matters inside recommendation systems, which decide what each customer sees first. If you only ever rank by popularity, you also risk popularity bias. And whenever you change how you rank, test the new order against the old one with real customers, as I describe in multi-armed bandit testing.
For consumer brands, product ranking is one input to the metrics leadership watches, like retention and lifetime value. You can see how we report those across seven brands in our consumer goods case study.
Talk to us
If you’d like a second opinion on how your site ranks products, the first call with one of our senior engineers is free and takes 30 minutes.