Engagement is essential.
For a lot of businesses, personalizing great content or products is the key to high customer engagement. For companies with lots of products where users cannot sift through everything on their own, a recommender system is an effective and valuable tool with which they can show the right item to the right user at the right time.
These recommender systems work well until the number of products skyrockets, or when products are changed and/or added very rapidly. In such conditions, relevant and personalized recommendations become much more challenging to deliver.
The best modern example of a company whose recommender system is faced with this challenge is YouTube, where 300 hours of video are uploaded every minute!
We’ve all experienced searching for a five-minute video and losing an hour or more to the YouTube rabbit hole. That’s not a mistake. The average mobile viewing session lasts more than 40 minutes, up more than 50% year-over-year.
YouTube gets engagement.
YouTube captures your attention by providing incredibly accurate recommendations, in near real-time. But how is that even possible with such colossal data rates?
YouTube has billions of potential videos to show you. Even the most efficient recommender algorithm cannot scale to billions of products and update frequently enough to incorporate all the new content. So YouTube did something smart; they created a two-step solution to the problem.
- For each user, in real-time, YouTube uses some high-level information about what types of videos were watched in the past to filter through billions of options and generate a few hundred candidate videos tailored to each user
- A very accurate recommender algorithm sorts through this narrowed-down selection of videos and ranks them according to what the user is most likely to engage with
You don’t have to look for any accuracy metrics here. YouTube recommendations are about as good as they get, and that’s shown by the incredible engagement numbers: Around 5 billion videos are watched on YouTube every single day by their nearly 30 million daily visitors!
Clearly, the two-step recommendation process works well for them.
This is generally how we approach “Big Data” in practice; we first look for ways to simplify or cull down the huge number of examples. This might include:
- Training our models on a short time window of data, removing the need for years of data collection that might only add a trivial amount of increased accuracy
- Performing intelligent segmentation that allows us to omit irrelevant items and create smaller, more manageable models
- Using special techniques to omit or combine multiple data fields
We often work with large datasets, but they usually don’t stay big for long; by employing some smart decisions about what’s important to the user and the company (good and fast recommendations that drive engagement), we often find that data science doesn’t have to be overly complex or incredibly computationally expensive.
If you’re running into capacity issues or are struggling to find ways to engage your customers with an increasingly large data set, contact me at zank@bennettdatascience.com and tell us about it. You may be a simple solution away from a better customer experience!
Read about the YouTube recommender here:
https://hackernoon.com/youtubes-recommendation-engine-explained-40j83183
And here are some mind-blowing stats from YouTube in 2019:
https://merchdope.com/youtube-stats/
Of Interest
A.I. ‘Outperforms’ Doctors Diagnosing Breast Cancer
A study in the journal Nature suggests that A.I. is more accurate than doctors in diagnosing breast cancer from mammograms. But before we start replacing all our doctors with iPads, there’s a lot more to treating a human than a simple diagnosis. This topic is surrounded by ethics and privacy and myriad other concerns. I believe that at best, A.I. is an important tool to assist physicians, not replace them. https://www.bbc.com/news/health-50857759
The Hidden Dangers in Algorithmic Decision Making
While we’re on the topic of A.I. making decisions for us, have you ever considered how the content you consume all day was chosen for /recommended to you? From YouTube videos to the latest Spotify playlist to your Instagram feed to the movies you watch at night… All of those sources use algorithmic recommenders to deliver your content. The problem is with so many of us in that loop of recommendations, these algorithms can generate a tremendous amount of bias for the content we consume. Read more about this fascinating and very real dilemma here: https://towardsdatascience.com/the-hidden-dangers-in-algorithmic-decision-making-27722d716;a49
A.I. is a Fast-Growing Field and That’s not Going to Change any Time Soon
A.I. jobs are on the upswing, as are the capabilities of A.I. systems. The speed of deployments has also increased exponentially. It’s now possible to train an image-processing algorithm in about a minute – something that took hours just a couple of years ago. Everything from A.I. conference attendance to the shrinking amount of time required to build and deploy intelligent models, speaks to the huge growth of A.I. https://www.zdnet.com/article/artificial-intelligence-the-score/