Reading Time: 5 minutes

by: Zank, CEO of Bennett Data Science

Eliminate the Extraneous: The Art and (Data) Science of Holding a ‘Conversation’

YouTube is one of the most successful user-generated video sharing platforms of the century. With over 100 million videos available, one might reasonably presume that the most important feature to YouTube’s success is the search engine. Users (that’s us) know what we want to watch, right?

The only snag is…we don’t! (Somewhat comforting when you think about the 2 billion people now singing baby shark). A recent study conducted by Zhou, Khemmarat, and Gao found that the primary source of views for the majority of YouTube videos is the video recommendation engine.

Baby Shark taking YouTube by storm. The internet is a strange place…

In layman’s terms, we’re talking about the bar on the side of the page that says, “Hey Zank, you seem to love 90’s rock. You’re going to love this clip of Pearl Jam.” In data science speak, this gets called a recommender algorithm, a recommendation engine, or simply a recommender. We’ll stick with the latter for the rest of the post.

These recommenders are everywhere, from YouTube, to Amazon, to Netflix. What distinguishes the good, the bad, and the just plain ugly is the notion of ‘conversation.’ In this post I’ll explain how to use a mixture of common sense, math, and of course…data (it is 2019, after all), to provide users with an experience that is conversational. Without further ado..let’s dig in!

Eliminating the extraneous: bringing back the lost art of conversation:

I think this is best explained from the perspective of a user. Let’s say we’re selling books and we have a very good book recommender. This recommender is an algorithm that models, or estimates, the likelihood you’ll be interested in certain books, based on your own tastes, purchases, browsing data, etc., as well as those of our other users.

We’re proud of the recommender we’ve built. It works well, and from all our initial tests, it appears to be showing books that will increase engagement of our users, leading to increased retention and increased sales.

Based on how we’ve scored (tested) our model against historical data, we’re doing a pretty good job of predicting the next thing the user buys. Now we’re ready to put this thing out the wild (deploy it: a fancy way of saying put it into our mobile app or web site).

Now it’s live. When users come to our site, they see books recommended just for them. No one else, unless coincidentally, would see the same books. We’re ecstatic, because

a) the recommendations make sense for the users we tested 
b) our users are happy, discovering authors and titles that are right up their alley and 
c) our retention and sales are skyrocketing. Win, win, win.

Win, win, win.

So let’s take a look at what a recommendation might look like, taken from my Amazon recommendations:

Hm…most of these are pretty interesting to me, but let’s say ‘The Power of Habit’ just doesn’t tickle my fancy (side note, it’s actually a fantastic book, I highly recommend it!). Notice there’s no way for me to express that I don’t like one/all/any of the books? I don’t have a way to interact with the algorithm, or to have a conversation with the ‘machine’ (in the least creepy, Her-esque way possible). Let’s fix that. We want something simple, yet interactive.

How about this?

Now, the addition of the thumbs up/down voting links makes it effortless to trim away books I don’t like or reinforce the recommendations I’m already receiving. To be conversational, the moment I use the thumbs down, the book should immediately vanish (because that means the user is being heard) and with a thumbs up, a message should let the user know that the site is listening.

It’s all one big conversation–an interactive dance of getting to know one another’s tastes and recommendations. I’m not picking on Amazon here, but I do think they’re missing an easy opportunity to converse with their users (consider how Spotify uses your feedback to fine-tune your playlists as a best-in-class example).

Although this feature doesn’t currently exist in Amazon, let’s look at how they do invite us into a dialogue. Amazon has an entire page devoted to this topic (find it here). The blue boxes are my own emphasis.

“You can tell us about items you like, even if you didn’t purchase it on Locate the item on, then write a review and add a rating. We’ll use that rating to help provide you better recommendations.”

This is very conversational and genius! Anyone who does this is giving Amazon information about products Amazon never knew about. And yeah, this will lead to much better recommendations. How can Amazon do this? Because they listen. Take a look at the page below.

I believe that the first step in any good conversation (online or otherwise) is to listen with the intent of understanding, not responding. 

Image result for listen

If a user is heard, and understood, then we as data scientists can take appropriate action. If, instead we don’t give the customer a method for thoughtful interaction, we lose twice; the customer loses confidence in our ability to interact with them, and they don’t have any way of helping us fix it or otherwise make our predictive algorithms smarter.

For previous purchases, there’s a strong “I wanted that” signal. Here’s how they allow conversations here, on an individual level:

On the far right, I can let Amazon know exactly how to treat this purchase. It sets the goodness of the recommendations as the responsibility of the user! Genius!

Why doesn’t Netflix have something similar? They do! When you watch 20 minutes then don’t binge-watch the next 36 hours of Breaking Bad, it’s an important signal to Netflix that you probably didn’t like the show. That signal gets stronger over time as you continue to not watch it, then fades away, as most preferences do, over longer time periods.

As this example shows, recommenders aren’t always overt. They may run behind the scenes, gradually learning your preferences and tailoring your experiences to cut out what is extraneous and serve up an additional serving of your favorite books, shows, songs, gizmos, and gadgets.

Online conversations

Online conversations like these are essential for customer satisfaction. They’re done well when one action leads to another, appropriate and immediate action. To learn how to bring the joy of recommenders to your site/app (and oh, so much more) fill out this simple three question form. The team at B_D_S will contact you within two business days to start our own …conversation (I had to).

Thanks for reading & talk soon!


Zank Bennett is CEO of Bennett Data Science, a boutique data science consultancy that works with companies from scrappy startups to Fortune 500’s. B_D_S specializes in transforming large volumes of data into answers, finding novel ways for companies to grow their products and revenue using data, while maximizing the effectiveness of existing data science teams.

Signup for our Newsletter