by: Zank, CEO of Bennett Data Science
I was asked an interesting question last week during a client meeting. It caught me a bit off guard because it was almost philosophical. Here’s the question:
“Where does intuition end and data science start?”
This is a fantastic question! It shows a desire to understand the continuum between expertise and analytics; to understand just how far expert opinion can take a company or product initiative. And that’s a very important transition to understand for small to medium-sized companies who are looking to transition into data-driven projects.
Subject Matter Experts
As an answer, I was fortunate to be able to use my wife as an example. She’s a designer, and quite a good one. You could call her an expert. We met when she was still in college and had almost no real-world design experience; Today, she’s got more than a decade of experience designing for clients like the NFL, Monster and a few celebrities. When I ask her how she approaches the web design process, she says,
“I get closely involved with each client, understanding their needs… Then I use that information to match their vision with the latest design trends. I don’t have a crystal ball to help me know exactly what users will like, but strong design sensibilities and an eye for white space are the best starting place we have.”
I love that explanation. Summed up, it says to me: seasoned professionals (or subject matter experts) are the best at setting a course when there’s no additional information to leverage. And that’s similar to how we see things unfold with data science.
Here’s a screenshot from her site:
Domain knowledge is essential for effective data science.
Through careful understanding of a product domain, we can employ a myriad of techniques to build powerful predictive elements. This domain-specific knowledge is like a key that unlocks our ability to add predictive power to our work. This is one of the reasons why data science is so difficult to replicate from one client to the next; rather each new product space requires careful consideration. Here are a few examples where we see domain knowledge all the time:
-
Feature engineering – this is where data scientists combine or deconstruct bits of data to build new, highly descriptive variables. For example, we may turn an ugly timestamp in to “weekday” or “weekend”. That’s a trivial (and often powerful) example, but domain experts are essential when building new variables for our models. Feature engineering is a big part of what wins Kaggle (a data science competition website) competitions; the competitors at the top of the leaderboards generally spend a lot of time understanding the domain to build new and informative features.
-
Building simple models – domain experts often build simple yet effective models based on heuristics. For example, a popularity-based product recommender, where everyone gets the same (non-personalized) recommendation of the top-5 most popular products, is just a simple count of what’s popular. That’s hardly data science, but can perform quite well! Combine this with some simple customer segmentation and keep the data fresh and this simple model can pass for personalization until something better comes along.
-
User-product interaction experience – this almost gets back to feature engineering above. It’s a nod to the ability of an expert in user-product interaction to just “know” what elements are most important to users at what time in their lifecycle. It’s this type of deep domain knowledge that informs the best data science models. This is one of the reasons we put so much importance on clear communication, without which we’d miss a lot of the important stuff.
Conclusion
So, Where does intuition end and data science start? I think there’s a handoff, attained through a lot of clear communication. This is part of why I consider data science a product asset and not (necessarily) a technology. I know, I know, there’s a LOT of math and computing, and technology. But it’s all done for the sake of product. The more data scientists understand about those products, the more effectively they can do their jobs.
Zank Bennett is CEO of Bennett Data Science, a group that works with companies from early-stage startups to the Fortune 500. BDS specializes in working with large volumes of data to solve complex business problems, finding novel ways for companies to grow their products and revenue using data, and maximizing the effectiveness of existing data science personnel. https://bennettdatascience.com