Tech Tuesdays: The Controversy of Black Boxes

A “black-box system” refers to a predictive model that produces an output with no way for the operator or end-user to understand what was done “inside” the box. Black-box (or unexplainable) models are more and more commonly used these days, from predicting what a customer wants next to understanding your voice when you’re yelling at Siri to give you the weather in Tecumseh tomorrow.

The reason data scientists employ these black-boxes is because, generally, ordinary machine learning models (think decision trees or regression models) can’t approach the complexity required to accurately perform hard tasks such as text, image or audio processing we rely on daily. Ordinary machine learning models are typically easy to explain, but come with the tradeoff of lower accuracy than neural networks or more complex A.I. So for the sake of accuracy, we sacrifice explainability.

But what’s wrong with that?

A lot, it turns out.

The problem with black boxes

Regardless of how well A.I. performs, there are times when explainability is just as important as the end result. If you receive a perfect recommendation for a dress to wear at a special event days after the event occurred, it might be important for the A.I. to explain why such a recommendation was made and let the user provide feedback so the algorithm can adjust itself. But explainability is important for much more than just trivially sorting out such a poor product recommendation.

Wikipedia hints at some of the challenges:

Explainable AI (XAI) refers to methods and techniques in the application of artificial intelligence technology (AI) such that the results of the solution can be understood by humans. It contrasts with the concept of the “black box” in machine learning where even their designers cannot explain why the AI arrived at a specific decision. XAI may be an implementation of the social right to explanation. XAI is relevant even if there are no legal rights or regulatory requirements—for example, XAI can improve the user experience of a product or service by helping end-users trust that the AI is making good decisions.

The importance of trust and explainability

At the heart of XAI is trust. Trust that an algorithm is properly representing our real desires or preferences, and doing so “for the right reasons”. Assessing when the “right reasons” are present can be tricky. For some industries, it is critical to understand how A.I. arrived at its decisions, and in some industries explainable A.I. can even mean the difference between life and death.

One of the largest offenders was IBM’s infamously “Watson for Oncology” that was created to summarize electronic health record data to help medical professionals but turned out to be a model that made unsafe treatment recommendations. Lives were at stake here. Explainability would have been a major key to understanding why Watson for Oncology was so far off in some of its diagnoses. In one case Watson suggested that doctors give a cancer patient with severe bleeding a drug that could worsen the bleeding! Knowing “why” would have been very helpful in troubleshooting such a horrific mistake. In 2017, after having spent $62 million, IBM shelved the project. (Reference)

Understanding WHY a model provides the answers it does is very important in increasing generalizability and reducing bias. If an A.I. model performs very well at a single task, understanding why it performs well can help analytics professionals to predict when or where it will fail. For example, if a model trained on a particularly poor demographic is reused and applied to others from a particularly wealthy demographic, it may not perform well. Understanding why a model works can be key to how well it will generalize to new populations.

Horrible bias can occur when models are trained without considering marginalized populations that may not be present in the training data. Thankfully, this is a topic receiving a lot of attention lately but it’s still a big concern that can be harmful and lead to much more damage than simple embarrassment for companies.

What can we do?

So, what can we do about this trade-off between the extremely attractive (although sometimes misleading) accuracy of black-box models and their inherent lack of explainability?

I believe the answer lies in using humans in the loop where necessary until there is sufficient data to remove people from the equation.

For example, A.I. currently surpasses human experts in breast cancer prediction, outperforming radiologists by more than 10% (Nature). But this result needs to be taken with caution. In this article, the author for example argues that the detection of cancerous or pre-cancerous lesions may be of little benefit to a patient who may never die from such legions. Adewole Adamson, a dermatologist and assistant professor at Dell Medical School defines this type of overdiagnosis as “Calling things cancer that, if you didn’t go looking for them, wouldn’t harm people over their lifetime.”

Want to see what Google is up to with XAI? Here’s their page on Explainable A.I. (in beta, of course) where they offer a number of tools and frameworks to help understand and interpret machine learning models. I’d love to hear what you think.

I hope you have a good week!