Explainable AI

Explainable AI is artificial intelligence that can show the reasons behind its outputs in a way people can understand and check. If a system gives a loan a risk score, flags a document or suggests a decline, an explainable system also says which factors drove that result.

The US National Institute of Standards and Technology (NIST) describes four principles for explainable AI:

1. Explanation. The system gives evidence or reasons for its outputs.
2. Meaningful. The explanation makes sense to the person receiving it, whether that is an underwriter, an examiner or a borrower.
3. Explanation accuracy. The explanation truly reflects how the system produced the output, not a tidy story written afterward.
4. Knowledge limits. The system only operates in the conditions it was designed for and signals when it isn't confident.

Why it matters for lenders

Lending law expects reasons. In the US, the Equal Credit Opportunity Act and Regulation B require a lender that denies credit to give the applicant specific, principal reasons in an adverse action notice. Those reasons must match the factors the lender actually considered. A vague line such as "did not meet our internal standards" is not enough. If a lender uses a model it can't explain, it can't write a truthful notice.

Explainability also supports fair lending reviews. To check whether a tool treats groups of applicants differently, a lender needs to see which inputs drive results. A factor that looks neutral, such as ZIP code, can stand in for a protected trait.

Funders, auditors and examiners ask the same question in different words: how was this number made? An explainable system, paired with an audit trail, lets the lender answer it.

Explainability is easier for simple models than for large language models, which often can't show their exact reasoning. That is one reason many lenders use generative AI only to prepare files and keep the decision with a person.

How it works in practice

A lender uses an AI tool that gives each small business application a risk flag. For one application, the flag reads "elevated risk." The underwriter opens the explanation. It lists three factors: deposits fell 30 percent over six months, the account had four returned payments, and existing debt payments use most of the monthly cash flow.

The underwriter checks each factor against the bank statements and confirms them. She declines the loan and uses those same, verified reasons in the notice, written so the owner can understand them. When the compliance team later tests the tool's flags across groups of applicants, it uses the logged factors to see what drives the results.