Need more data? Simulate it with Monte Carlo methods

Don’t have enough data for a new AI project? You can often simulate it. In this post, I explain how Monte Carlo simulation works, and how we use it to decide which data is worth collecting before a client spends money collecting it.
How Monte Carlo simulation works
In a Monte Carlo simulation, we make assumptions about the kind of data we have and how it’s spread out. Then we let a computer “play the game” over and over, generating data as it goes.
Card games show why this is useful. In solitaire or multi-deck blackjack, there are so many possible hands that working out the odds by hand quickly becomes impossible. A simulation can play thousands of hands with a given strategy and tell you how often it wins. You could use it to work out exactly when to hit and when to stand.
Uses outside the casino
Elections. With polling results, past turnout by county, and a few other details, you can simulate an entire national election thousands of times. Forecasters do exactly this. The same approach works for sports, weather and customer purchases.
Product recommendations. Recommenders can use a lot of customer data. But how much is enough, how useful is each piece, and how well will the algorithms use it? Questions like these are why I simulate customer data on most recommender projects. Clients rarely come to us with complete customer data. When a client wants to know whether collecting one more field is worth the effort, we simulate that field under clear assumptions and watch what it does to the results. That helps the client choose which data to go after. There’s more on recommenders in a guide to recommendation systems for business.
Simple idea, careful setup
Simulations are easy to understand and hard to set up well. The results are only as good as the assumptions behind them, so the assumptions need to come from people who know the business and the data. That’s part of the larger job of getting data ready, which I cover in why so much of an AI project goes to cleaning data.
If you’re weighing whether your data can support an AI project, that’s exactly what our fixed-fee Discovery answers before anyone builds anything. You can read about it in how we work with clients.