Five Questions To Ask Your AI Vendor About Explainability

Subscribe to Our Blog

A wave of new FinTech companies is pushing traditional lenders to consider new technologies to stay competitive. One of the most promising technologies is advanced machine learning (ML), a branch of AI that uses mountains of data and sophisticated math to help lenders make better decisions and predictions.

Before making any investments in ML, lenders need to ask the right questions in order to capture the gains. Take the issue of explainability. By law, lenders must be able to show why they approved or denied a loan. In the old days (and we’re still in them) that was easy: Typical underwriting models have only 30 or so variables, and none of them interact with each other, making it clear which ones are driving the decision. But ML models analyze thousands of factors and the interactions between them, making it all but impossible for humans to determine easily which variables are driving the decision.

This black-box problem has attracted a handful of techniques (such as LIME, permutation impact, and leave-one-column-out) that claim to peer into ML models and extract the top reasons for loan approvals or denials. Essentially, these methods take a very complex model and pretend it’s easy in order to explain it. But in recent tests, we found these methods lacking in ways that could expose you to risk. None of them performed as well as the explainability baked into ZestFinance’s ZAML software, which uses a different explanation method, derived from competitive game theory and multivariate calculus, that generates explanations from the actual underlying model.

To capture the benefits of ML, it’s crucial that banks and lenders looking to switch to ML underwriting retain the ability to get consistent, accurate, fast explanations from credit models. Here are five key questions to ask any AI software vendor before you make any decisions about deploying ML in your business.


Can your explainability technique run and chew gum at the same time?

Most explainability techniques can analyze the effect on a model of only a single variable at a time. With hundreds of variables in an ML model, that kind of brute force work can drag on for hours or days. And without capturing the interactions between and among multiple variables, something most explainability techniques are incapable of doing, your explainer will compromise on accuracy.


Does your technique get under the hood?

Most explainers hazard a guess as to what’s under the hood of an ML model by looking at what data goes in, and what comes out. They don’t examine the model’s internal structure to understand the interactions between the many variables. That means you’re getting much less information than with explainers that delve into the model. You might be gambling with accuracy.

An example that shows why this matters: An explainer that looks at a only a section of a model’s data space might fail to see bias or discrimination and conclude that no disparate impact exists, which may be true. But because you’re only looking at a small section of the model, there is no guarantee that disparate impact doesn't exist somewhere else in the complexity of the real model. Missing such an important detail due to a lack of accuracy in your fair lending analysis raises serious ethical and regulatory risk once the complex model is deployed to make real-world decisions.


Is your explainer playing make-believe?

You need to know if your explainer is looking at the final model because that’s the only one that matters. You might get some insights from analyzing works in progress, but again, you’ll be compromising on accuracy. Some techniques also probe the ML model with non-real data. But you don’t want an explainer that uses synthetic or randomly shuffled data, because that leads to inconsistencies.


Is this explainability technique a consistent performer?

Consistency is key for underwriters. Every time a borrower application is scored, a model should yield the same prediction, and if there’s a denial, it should yield the same set of primary reasons for the denial. Every explainer used for lending should return the same answer when the input is the same. If you don’t have that, you might get different explanations from analyzing the same borrower twice. For example, if you re-build your model using the same data, your explanations should remain the same. This seemingly trivial property is not true for most explainability techniques.


Does your explainer see the bigger picture?

Underwriters need their ML explainability technique to more holistically understand a model, which explainers looking at local “slices” of a model often fail to do. Let’s use a basketball metaphor: if you were trying to explain to a Martian why Kobe Bryant is a great basketball player, you’d probably mention that he’s taller than most humans. But an explainer such as LIME, which gets its answer by looking at a small piece of the model, might see a lot of other NBA players, who are also going to be tall, clustered around Kobe and thus determine that his height is not so much of a differentiating factor when indeed it is, compared to typical humans.


If your vendor can’t provide a clear, unequivocal “yes” to each of these questions and explain how their software meets these criteria, it may be time to look for a new AI vendor. At ZestFinance, we built ZAML specifically to ensure that you’ll have access to the clearest, most accurate explainability on the market. Read more about ZAML here.

New call-to-action