← Blog

AI models need maintenance: how to keep them accurate

A predictive model is like a magic eight ball. You put some data in and an answer comes out. Unlike the eight ball, a model needs fresh data from time to time, or its answers slowly go wrong.

I first wrote about this in 2019, after a colleague asked me a simple question: how long does a model last? The answer still shapes how we run every system we put into production.

A simple example

Say an amusement park wants to know how many umbrellas to stock for a given amount of rain. We collect a year of data on rainfall and umbrella sales and fit a line through it. That line is the model. Give it a rainfall forecast and it tells you how many umbrellas you’ll sell.

That model works well as long as conditions stay the same. Then winter comes.

Chart of umbrellas sold against rainfall in summer and in winter, with two separate trend lines showing that more umbrellas sell in winter for the same amount of rain

In winter, visitors buy more umbrellas for the same amount of rain. Cold and wet feels worse than warm and wet. A model trained on summer data will undercount every winter day. The data in this chart is made up to show the point, and the point is real: the first model was right until conditions changed.

Change is the normal state

Your customers change all the time. Some changes are slow, like shifts in what people spend money on. Some come every year, like holidays and seasons. Some come out of nowhere, like a recession or a news event that changes what people buy overnight.

A model only knows what was in the data it learned from. If something new happens, the model has no way to react to it on its own.

How we keep models accurate

We retrain on fresh data. Depending on the business, a model might be retrained every week, every night or every time new data arrives. Nightly is common. Retraining keeps up with new customers, new products and changing tastes.

We plan for the changes we can see coming. For seasons and holidays, we give the model several years of history and tell it what time of year it is, so it learns that December looks different from June.

We react quickly to the changes we can’t see coming. When something big and sudden happens, we can shorten the history the model learns from to the last few weeks. That puts more weight on how customers behave now and less on how they used to.

We watch the data and the results. We monitor the data coming in for big shifts, like a very cold summer, and we compare the model’s live results with what it promised in testing. When they drift apart, we find out why.

We look for better data. Often the biggest gain comes from adding a new source of data, like weather or seasonal trends, that the original model never had.

A real model has hundreds or thousands of inputs. My umbrella example has one. The work scales up, and the principle doesn’t change.

What this means for you

When you put a model into production, plan for someone to look after it. I always recommend keeping the people who built it involved after launch. I go through what that looks like in putting AI into production, and a model nobody maintains is one of the five reasons AI projects fail to pay off.

Good data makes maintenance much easier. If your data takes weeks to clean every time you retrain, start with getting your data ready for AI.

This is how we work with clients. At Irth, we have run more than 70 models in production since 2020, and we still operate them. You can read about them in the damage-risk models we built for Irth. For a Fortune 250 consumer goods company, we have built and operated the analytics platform for seven brands since 2019.

After go-live, you choose who looks after the system. Our engineers can run it, we can run it together with your engineers, or your team can run it with us on call. The Bennett ONE questions and answers explain the options, and in every one, Bennett ONE reports each result against the value it started from, so you can see when a model starts to slip.

Talk to us

If you have a model in production and nobody is sure how well it’s holding up, the first call with one of our senior engineers is free and takes 30 minutes.

Let's talk about what you want to improve

Talk to an engineer