How popularity bias undermines personalization

Popular products sell well because they’re popular. Recommending them is a sensible place to start. The trouble starts later, when best sellers are all your customers have ever been shown and your data can’t tell you anything else. That’s called popularity bias, and it can hold back your personalization for years.
I first wrote about this in 2019. This version also covers the simple ways to personalize that I recommend before any advanced modeling.
Start with popularity
When you don’t know much about your customers yet, showing them popular products works. Count how many times each product was bought, and show the top few. That’s much better than picking products by hand, which doesn’t scale, and far better than picking at random.
It isn’t personalization, though. Everyone gets the same email with the same five products.
Simple ways to personalize
You can treat different customers differently long before you build anything advanced. All three of these only need counting.
Ask good onboarding questions. Before customers shop, ask them a few questions. The best questions split your customers roughly in half. If everyone gives the same answer, the question tells you nothing.
Keep separate best-seller lists for different groups. Look at what you already know, like age, location or what people told you during onboarding, and check whether those groups buy differently. If they do, keep a separate popularity list for each group. If your customers aged 18 to 25 buy different things from your customers over 40, that’s two lists, and almost no extra work.
Recommend what was bought together. Say a shopper looks at a red cashmere sweater, black jeans and a grey t-shirt. Find everyone else who looked at those same items, see what else they looked at, and show those products. This is called market basket analysis, and it comes from studying what shoppers put in the same grocery basket.
Use the same groups and lists in your marketing emails. Sending everyone the same message means it speaks to almost no one.
How popularity bias creeps in
After you’ve recommended popular products for a while, your data shows that your customers like popular products. Of course it does. That’s all you showed them. The popular products get even more popular, and the cycle feeds itself.
The real damage shows up when your team tries something smarter. A new recommendation model gets judged against your history. Your history is full of popular products, so the new model, which recommends other things, looks worse than it really is. It seems like nothing beats popularity, when the data was never a fair test.
Two ways out
Avoid global popularity from the start. The separate lists by group are my favorite way to do this. As long as the groups really do buy differently, you avoid most of the bias, and the recommendations usually get more relevant too.
Reduce popularity slowly. You can’t judge a new model fairly on biased history, so test it live. Show the new model to some customers, compare it with the popularity list, and let it take over as it wins. Over time, your data fills with the new model’s recommendations and the old popularity data matters less. I explain the testing in multi-armed bandit testing.
The new model shapes your data too, of course. The goal is a model that shows people a wider, more personal range of products.
Where this leads
The end goal is one-to-one personalization, where each customer sees products picked for them. That’s what a recommendation system does, and it works best on data that isn’t dominated by best sellers. It also helps to rank products by rating correctly, so a product with two good reviews doesn’t jump ahead of one with hundreds.
Personalization pays off when you can see its effect on the metrics that matter, like retention and lifetime value. For a Fortune 250 consumer goods company, we built one analytics platform for seven brands that shows lifetime value, acquisition cost and retention for every brand on the same definitions.
Talk to us
If your recommendations still lean on best sellers, the first call with one of our senior engineers is free and takes 30 minutes.