← Blog

Lying with statistics: how to spot a misleading figure

Some statistics are built to alarm you more than inform you. This post shows two common ways a true figure can mislead, and the questions to ask before you act on one, whether it’s in the news or in your own company’s dashboard.

“It doubled”

A classic example is a news report that cases of something have doubled in a town. It’s worth remembering that twice a small figure is still small, and twice a large figure is still large. Going from two to four is a doubling. So is going from one billion to two billion.

For some reason, most people hear “doubled” as a very big deal, which is exactly why reporters use it. The word tells you much less than it seems to.

The next time you hear it, ask for more:

  • What were the before and after counts? Going from 10 cases to 20 may not be news at all.
  • Who was affected? In health data, age is often a major risk factor, so the age of the people involved changes the story.
  • How big is the population? The same increase means very different things in a small town and a large city.

Also ask how the data was collected. Two places can count the same thing in different ways. What one region records as a death from a disease, another might attribute to a different cause.

Context is everything

Misleading statistics are easy to produce with one technique: zoom in very far and describe what you see without context. There’s a whole book about this, Darrell Huff’s How to Lie with Statistics, first published in 1954.

One of Huff’s examples: during the Spanish-American War, the death rate in the Navy was nine per thousand, and for civilians in New York City over the same period it was sixteen per thousand. Navy recruiters used those figures to argue that it was safer to be in the Navy than out of it.

The two groups weren’t comparable. One was chosen for youth and health. The other included everyone, including babies, the elderly and the sick. A fair comparison would have looked at New Yorkers of the same age as the sailors.

The same thing happens inside companies

This isn’t only a problem in the news. Business dashboards zoom in too. A conversion rate that “doubled” last week might rest on a handful of orders. A sales region that looks like a star might simply serve a different kind of customer. That’s why we test changes properly before we trust them. I explain how in better testing equals more revenue.

The takeaway

When a statistic makes you feel something strongly, ask whether it’s been taken out of context from a zoomed-in view designed to win your attention. Many headlines deserve a longer story and much more context.

Context is also why every result we report at Bennett sits next to where the metric stood when we started. Bennett ONE stores every result with its source, so anyone can check it.

Let's talk about what you want to improve

Talk to an engineer