Sometimes several small solutions perform better than one big one.
This week I want to show you why it’s worth considering replacing one big predictive model with multiple smaller ones. This is something we’ve been doing for our clients lately and has generated a lot of value, allowing them to get more out of their data science spend.
Let’s say we deliver a model that predicts churn based on customer attributes and actions. This is rather standard. Such a model might trigger an automated email or even a phone call from the retention department. We see and work on this type of model all the time.
Now imagine that we have one big model that takes as its inputs all sorts of metrics such as age of customer, number of website visits in the past week/month/quarter, recent spend, etc. As an output, the model reports the probability that the customer will churn in the next seven days. This might be something like: there is a 27% chance that the given user will churn in the next seven days.
Overtime, we can assess the accuracy of such a model. Let’s say our model is correct 62% of the time. That’s better than a coin flip, but still not fantastic.
Then let’s say we have a realization that demographics matter to churn. Maybe customers in colder climates churn more often than those in a warmer climate, for example. If there are three different regions of varying climate, it may be appropriate to build three different models; one for each climate condition.
In this manner, when a request to assess a customer’s churn is received, it is first routed to the model best suited to the climate where the current customer lives.
This sort of switching is very common in machine learning but is often overlooked by practitioners who build more-complex models in the attempt to keep things unified and simpler. Often we find that breaking models out like this can yield big gains with the side effect of helping us learn about the behaviors of various groups within a collection of customers.
The take-away: It’s good to remember that there are many ways to a solution and sometimes breaking things apart can actually provide a simple solution to a complex problem.
I hope this helps. Have a great week!
Attacking Machine Learning with Adversarial Examples
Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake; they are like optical illusions for machines. In this post on OpenAI, Ian Goodfellow et al. shows how adversarial examples work across different mediums and discusses why securing systems against them can be difficult.
Why Data Integrity is key to Machine Learning Monitoring
Machine learning applications are built on data: large amounts of data are streamed into ML systems in order to train models on historical examples and make high-quality predictions in real-time. Yet many machine learning projects degrade or fail, in large part because data integrity is difficult to maintain. This article by Towards Data Science discusses why data integrity is key to machine learning monitoring and how to catch data integrity issues.
Data Science and the Art of Persuasion
Data science is growing up fast. Over the past five years companies have invested billions to get the most-talented data scientists to set up shop, amass zettabytes of material, and run it through their deduction machines to find signals in the unfathomable volume of noise. It’s working—to a point. Data has begun to change our relationship to fields as varied as language translation, retail, health care, and basketball. But despite the success stories, many companies aren’t getting the value they could from data science. Even well-run operations that generate strong analysis fail to capitalize on their insights. Efforts fall short in the last mile, when it comes time to explain the stuff to decision makers. Read more about data science and the art of persuasion here.