← Blog

You probably don’t have a big data problem

Bennett Data Science Tech Tuesdays Big Data

As companies grow, they create more data, and processing it gets more expensive. This post shows how to handle large amounts of data without paying for massive cloud computing, using a trick YouTube uses for its recommendations.

How YouTube handles millions of videos

One of the largest recommendation systems on the planet is YouTube’s. Imagine recommending personal videos to millions of people while new videos pour in every minute. Feeding all of that data into one giant recommendation engine would fail, and even if it worked, it would cost a fortune.

So YouTube does something clever that you can do too. It first narrows the full library down to a few hundred videos you might be interested in, and only then ranks those videos with its recommendation engine. YouTube calls the first step “candidate generation.” The engineers describe it in Deep Neural Networks for YouTube Recommendations.

The pool goes from millions of videos to hundreds, then to dozens. Data scientists can work with hundreds of things to rank. Millions are a lot harder, especially with as many viewers as YouTube has.

The same idea at a smaller company

Few of us will ever work at YouTube’s scale, but the idea of filtering first applies to smaller companies with far less data.

Say a company wants to search tens of millions of documents for relevant content and also judge the tone of each author. That’s within reach of plenty of mid-sized companies, and processing every document can get expensive. A few simple questions can shrink the pile before any expensive processing starts:

  • How old is each document? If freshness matters, maybe only 30% of the documents are relevant.
  • Does length matter? If so, how many documents are long enough?
  • Does the type of content matter? If you have that information stored with each document, the list gets shorter again.

I like to say that I don’t often work with “big data.” When we apply algorithms to a specific problem, the right data is usually fairly small, just as it is for the YouTube recommender.

What to do on your next project

The next time a project seems impossibly large or more expensive than you’d like, look for your own version of candidate generation. Make sure your expensive computing only runs on the most relevant data.

If you’re working on recommendations, I go further in recommenders and business. And if data volume is holding up a project, our engineers can look at it on a free first call.

Let's talk about what you want to improve

Talk to an engineer