← Blog

Why explainable AI matters, and how we build it in

Bennett Data Science Tech Tuesday Explainable Artificial Intelligence

A “black box” is a model that gives you an answer with no way to see how it got there. Most of the AI people use every day works this way, from the recommendations on a shopping site to the language models behind today’s AI assistants.

I first wrote about this in 2020 and 2021. Back then the black boxes were recommendation engines and image models. Now companies are asking AI to draft contracts, answer customers and approve orders, and the question of why it gave an answer matters far more.

Why companies use black boxes

Simple models are easy to explain. A model that predicts a home’s price from its size is two values and a line, and anyone can follow it. I walk through that example in putting AI into production.

Simple models can’t read a contract, understand speech or write an answer to a customer’s question. The models that can are far more complex, and they’re much harder to explain. For years, companies traded explainability for accuracy because the accurate models won.

What goes wrong without an explanation

Some mistakes are harmless. If a site recommends a dress for a party that ended three days ago, the right fix is for the system to explain why it picked the dress and let the shopper correct it.

Other mistakes are serious. In July 2018, STAT reported that internal IBM documents showed Watson for Oncology had recommended “unsafe and incorrect” cancer treatments. STAT also reported that the system had been trained on a small set of hypothetical patient cases. When a system can’t show its reasoning, nobody can see that kind of problem until it reaches a patient or a customer.

Knowing why a model works also tells you where it will fail. A model trained on one group of customers may perform well on them and badly on a different group. If you understand what the model relies on, you can predict that before it happens. Bias works the same way: when a group is missing from the training data, the model treats that group badly, and an explanation is often the only way to find out.

Keep a person in the loop

My view hasn’t changed since I first wrote this. Where the stakes are high, a person should make the final call until there’s enough evidence to trust the model on its own.

That’s how we built Bennett ONE. We assume every AI model will sometimes be wrong, confused or overconfident, so we design for it:

  • Every answer shows its source. Each answer links back to the document, table or record it came from, so anyone can check an answer in one click.
  • A person approves anything consequential. Any action that moves money, changes a record or reaches a customer waits for a person on your team. You can see where that step sits in how Bennett ONE works.
  • Every step is recorded. Every question, answer, approval and result is stored with its source, so you can always see what happened and why.

As I put it on our Bennett ONE page, “We do not ask an agent to behave well. We remove its ability to do the wrong thing.” The model can only recommend, a person decides, and the record shows both. You can read more about that choice in why Bennett ONE works the way it does.

The same thinking applies to older models. At Irth, our model flags the riskiest call-before-you-dig tickets so field teams know where to go first, and each operator chooses how much of that risk to inspect. You can see how that decision works in the damage-risk model we built for Irth.

What to ask your own team

If you’re putting AI in front of customers or letting it act on your behalf, ask three questions:

  • Can we see the source behind every answer?
  • Who approves an action before it happens?
  • If something goes wrong, can we trace exactly what the system did and why?

If the answer to any of them is no, fix that before you expand the system. I cover more of the questions to settle up front in why AI projects fail to pay off.

Talk to us

If you want AI you can check, the first call with one of our senior engineers is free and takes 30 minutes. We’ll tell you plainly whether Bennett ONE is a fit.

Let's talk about what you want to improve

Talk to an engineer