Tech Tuesdays: Why a Shocking 80% of Data Science Projects Goes to Data Cleanup

As discussed in one of our previous Tech Tuesdays on Deployment, a successful data science project generally comprises twelve essential steps. And these twelve steps can be roughly divided into three stages. Here’s the one diagram I come back to time after time that offers an overview of these three stages. It’s so simple:

And here’s how much time we spend on each section:

As you can see, the great majority of time is spent on data engineering. A significant 80%! It is important to point this out as the value of data engineering is often underestimated. And that’s exactly what I’m going to address in today’s Tech Tuesday. We’ll look at how a typical project might go and deconstruct the huge data engineering time sink.

The Value of Data Engineering

Machine learning is easy. If it weren’t, companies like Amazon and Google wouldn’t be so successful at automating it. That’s why the Machine Learning stage generally accounts for only 5% of time spent on a data science project. It’s simply not that difficult anymore.

Deployment (making machine learning insights available to a wide audience) is very similar, although not at all trivial. If we’re going to deploy (make use of) some fancy predictive algorithm, we need to feed it good data. Trash in, trash out, definitely applies here. And this is where the value of data engineering comes in.

Here, data engineering is the process of making the data usable for analytics processes. There’s a lot of cleaning and organizing that goes into the deployed machine learning model. Siri doesn’t just turn your lights on by magic! It took a lot of data engineers a lot of time to clean and organize data to train it to understand the human voice. “A lot of time” is an understatement.

So, if you want to streamline your data science projects, focus on data and hire appropriately, because the other parts are not nearly as difficult, expensive, or time-intensive.

Trifacta, a company that tries to make the data cleaning process easier, has this to say:

80% of the time spent on data analytics is allocated to data munging, where IT manually cleans the data to pass over to business users who perform analytics. Data munging is time-consuming and disjointed process gets in the way of extracting true value and potential from data.

That’s right, it definitely does get in the way.

But it’s not as straightforward as passing your messy data into a black box and out pops perfectly clean usable data. This is because each new dataset is different from the last. Customer interaction data will have different cleaning needs than customer profile data purchased from a third party, or analytics data gathered from a Google API.

This is especially true as more and more companies need to go beyond internal datasets for data enrichment. Data enrichment can be a differentiator as companies look to deeper user profiles for powerful profile or segment detection. But it comes at a cost: integration with your current data stack. How would you handle 3,000 more user attributes for each of your users? What if the data you purchase has inconsistencies? The answer is, it’s complicated and very time-consuming.

There’s no end to the snafus we see in datasets. From the simple zip code issue where zip codes are stored as integers (there are not integers!) to columns containing single values in some fields and large JSON objects in others! We’ve seen so many different versions of “dirty” data and each requires its own approach to handle, often requiring the involvement of stakeholders to help us clean it all up.

The take-away: Plan on spending 80%+ on getting your data into a workable format for effective data science. It’s a pain, sure, but when it’s done right, the big gains of deployment are only a short sprint away.