Stop 18 · Evaluation, Safety & Responsible AITopics · 7/9

Explainable AI (XAI)

In short

'The model said no' isn't good enough for a loan, a medical flag or a hiring decision. Explainable AI tools show which inputs pushed a prediction up or down, and by how much.

Techniques (SHAP, LIME, attention visualization) for understanding why a model made a specific prediction — important for trust, debugging and regulatory requirements in sensitive domains.

The blob sits across a bank desk while a friendly officer points to a simple bar chart on a sheet of paper.

Think of it like this

Asking the bank why

When a bank rejects your loan, you want reasons: 'Income too low for this amount, two late EMIs last year.' You can then fix something. SHAP and LIME give a model's prediction the same kind of reasons, one input at a time.

Watch it happen

SHAP values for one loan application (example)

base 0.30

income -0.10

late EMIs +0.25

tenure -0.05

final 0.40

1/5Start from the base value: the average predicted default risk across all applicants, 0.30.

The big ideas

What you need to know

Local explanations

Why did the model predict this for this one person? SHAP waterfall plots and LIME answer this.

Global explanations

Which features matter most across all predictions? Summary plots and permutation importance answer this.

SHAP adds up

SHAP splits a prediction into per-feature contributions that sum to the output, based on game theory.

Explains the model, not the world

A feature with high SHAP drives the model's output. That doesn't prove it causes the real outcome.

See it in code

Walk through the code

python

1import shap2from sklearn.datasets import load_diabetes3from sklearn.ensemble import RandomForestRegressor4 5X, y = load_diabetes(return_X_y=True, as_frame=True)6model = RandomForestRegressor(random_state=0).fit(X, y)7 8explainer = shap.TreeExplainer(model)9sv = explainer(X.iloc[:100])10 11shap.plots.waterfall(sv[0])   # why patient 0 got their score12shap.plots.beeswarm(sv)       # which features matter overall

1/4Train a normal model on a small built-in dataset.

SHAP on a tree model. Run it in a Jupyter notebook to see the plots. TreeExplainer is fast for tree-based models.

Save yourself the pain

Common mistakes

Oops

Saying 'feature X causes the outcome' because its SHAP value is large.

Do this instead

SHAP explains the model's behaviour. Causation needs experiments or causal methods.

Oops

Explaining only the overall model and never a single decision.

Do this instead

People affected by a decision want their own reasons. Provide local explanations for individual cases.

Your mission

Explain one prediction

~25 min

Quick check

A SHAP waterfall shows base value 0.30 and contributions +0.25, -0.10, -0.05. What's the prediction?

Go deeper

1Use SHAP or LIME to explain individual predictions from a classical ML model.
2Visualize attention weights to interpret a transformer model's focus on an example.
3Learn which domains (healthcare, finance) make explainability a hard requirement, not a nice-to-have.