When a marketing spike breaks your AI models

Consistency in marketing matters a lot to AI. In this post, I explain what happens to your models when a big campaign brings in customers they’ve never seen, and how to check the fit before you spend the money.
A familiar story
Here’s a common scenario, exaggerated to make the point. Your company steps away from its usual marketing plan to work with a celebrity who has a huge following. A post goes out, one of your products is shared everywhere, and new leads jump 20%. It’s thrilling.
At the end of the day you sit down with the results. Leaving out the vanity metrics, like total visits or clicks that never turned into a purchase, the results are worse than expected. There was so much traffic, and the celebrity is famous around the world. What went wrong?

We see this all the time. Let’s look at why it happens, how to see it coming, and how to get the most out of new traffic. For this example, success means more product sales.
Models assume your customers stay the same
Every system built on data assumes the underlying data doesn’t change much. Two assumptions matter here.
Customers don’t change much over time. Slow drift is normal. I mean big, sudden jumps. Imagine every one of your customers won the lottery on the same day. Would they buy differently? Probably. Would your recommendations still fit them?
Products don’t change much over time. A pencil doesn’t turn into a smartphone overnight. Products change step by step.
A big campaign can break the first assumption in a single day.
Three scenarios
The best case. The new customers are just like the old ones, and your systems handle them fine. The new traffic simply scales up your business, as long as nothing breaks under the load.
The middle case. The new customers are somewhat different. Say a whole group of them wakes up 14 hours before the rest of your customers. All the work you did to time your messages may no longer apply, even after adjusting for time zones. Shipping, communication style, and how similar they are to your current customers all need a fresh look. A new group of customers can be very good for the business, but you have work to do before they bring the same return as your existing customers.
The worst case. The new visitors came for the celebrity. They click around once or twice, buy almost nothing, and leave. The business pays to handle a wave of visitors who are about to disappear, the celebrity’s endorsement loses some weight, and your engineers spend the weekend answering alerts.
The good news is that the worst case can usually be prevented by asking the right questions up front, and by not getting star-struck. I haven’t yet seen a data tool that stops an executive from falling for the celebrity of the moment who can “send our business to the moon.” I’ve seen it more often than I’d like to admit.
Know your current and future customers
Before you go after a new audience, ask the people building your AI for a clear profile of your current customers. They understand your customers in the data better than anyone. From there, you can work together on how far marketing can go from that profile before your systems stop working well.
Age is an easy example. Say you’ve sold to young adults for two years. Someone suggests a celebrity whose reach could send hundreds of thousands of potential buyers to one of your products. A quick look at the celebrity’s following shows it’s mostly retirees. That sounds like an obvious no, but we’ve seen it happen more than once. These groups will respond to your messages and products very differently.
How to avoid wasted marketing spend
- Identify the segments of your current customers and describe each one carefully. We do a lot of this for our clients.
- Use those segments to build customer profiles based on the data. Some will show higher lifetime value, some buy more often, and some bring in new customers by word of mouth.
- Find out who the new channel or celebrity actually reaches. That information is often available on request.
- Compare your customer profiles with that audience and decide how much the campaign is really worth to the business. If there’s no overlap at all, walk away.
When there’s some overlap and the decision is hard, ask a few more questions:
- Will the boost last long enough to justify the work of serving these new customers well?
- Is it a good fit for your business?
- What will the lifetime value of these new customers be?
Customer analytics like this is a big part of the analytics platform we run for seven consumer brands, where leadership compares lifetime value, acquisition cost and retention across brands. If new customers do arrive, plan for your models to need retraining. I explain why in models need ongoing maintenance, and how to keep customers engaged once they arrive in how to retain and engage your customers.
If you’d like help checking a campaign before you commit, the first call with one of our senior engineers is free and takes 30 minutes.