← Blog

Why machine learning isn’t always the answer

Bennett Data Science Tech Tuesdays Machine Learning Isn't Always the Answer

A lot of teams use machine learning too early, because it’s the tool they have. This post covers when a simpler method beats a model, why junior teams fall into this trap more often, and what to check before you build anything complex.

When the only tool you have is a hammer

The law of the instrument applies here: when the only tool you have is a hammer, everything starts to look like a nail. Many teams use machine learning too soon, and under the wrong assumptions, because that’s the tool they have. After all, a data science team is supposed to produce predictive models, right?

There are plenty of good reasons to hold off. Maybe there isn’t enough data yet. Maybe the data isn’t the right type for the method. Maybe a simpler method would give similar results. In these cases, a simple rule of thumb can do the job well before machine learning is worth considering. That sounds obvious, but teams often jump in and throw complex algorithms at a problem before it’s warranted.

Google’s own Rules of Machine Learning start with this one: “Don’t be afraid to launch a product without machine learning.” Eugene Yan makes the same case well in the first rule of machine learning: start without machine learning. I’ve been saying and doing this for years.

Experience makes the difference

In my experience, veteran machine learning engineers reach for a model too early much less often than junior data scientists do. That’s a solid argument for hiring people with years of real-world experience. Schools can’t really teach when not to use machine learning in an analytics program.

So with junior data scientists, the emphasis has to be on doing the right thing for the problem in front of them. Pulling out a fancy algorithm and getting frustrated when it doesn’t work is a waste of time. Worse is tweaking the data or the model until it appears to work. I did that last one often when I started out.

A cautionary example

During the pandemic, researchers built hundreds of AI tools to help hospitals diagnose and manage COVID-19. MIT Technology Review looked at the results in hundreds of AI tools have been built to catch covid. None of them helped. Its conclusion: “None of them made a real difference.” Much of the failure came back to poor data. Cases were measured differently in different places, and a lot of data was gathered in a hurry with no checks in place.

Unrealistic expectations push teams to use these tools before they’re ready.

What to do before you build a model

Before rushing in with algorithms, understand the strengths and limits of your data. Then apply machine learning only where it’s appropriate. A few questions help:

  • Do we have enough examples of the outcome we want to predict?
  • Is the data consistent and reliable enough to learn from?
  • What would a simple rule get us, and how much better would a model need to be to justify it?

If you’re short on labeled examples, that’s a data problem first. I wrote about it in why so much of an AI project goes to cleaning data.

At Bennett, this question gets answered during Discovery, before anyone builds anything. Our engineers measure where your metric stands today and tell you plainly whether your data can support the result. You can read the steps in how we work with clients.

Let's talk about what you want to improve

Talk to an engineer