Stop 13 · Business Communication & StorytellingTopics · 4/7

Explainable AI (SHAP, LIME Overview)

In short

When a model rejects a loan, someone will ask 'why?'. 'The model said so' won't do. SHAP and LIME show which features pushed one particular prediction up or down.

Techniques for explaining an individual model prediction — SHAP and LIME estimate how much each feature contributed to a specific prediction, crucial for trust in high-stakes decisions (credit, healthcare, hiring).

The mascot sits at a dhaba table with three friends, pointing at plates to split the bill.

Think of it like this

Splitting the dhaba bill

Four friends share dishes and the bill is ₹1,200. Splitting it fairly means working out how much each person's orders added. SHAP does this for a prediction: it splits the gap between the average prediction and this one among the features.

Watch it happen

One loan's default risk, explained

Base 0.20

Income -0.08

Late pays +0.25

Age +0.02

Loans +0.11

Final 0.50

1/5Start at the base value: the model's average predicted default risk across all applicants.

The big ideas

What you need to know

Local vs global

Local explains one prediction. Global shows which features matter most across all predictions.

SHAP adds up

Based on Shapley values from game theory. Each feature's share plus the base value equals the prediction.

LIME zooms in

LIME nudges one example many times and fits a simple model nearby to see which features matter there.

Not cause and effect

Explanations describe what the model uses, not what causes outcomes in the real world.

See it in code

Walk through the code

python

1import shap2from sklearn.datasets import fetch_california_housing3from sklearn.ensemble import RandomForestRegressor4 5X, y = fetch_california_housing(return_X_y=True, as_frame=True)6model = RandomForestRegressor(n_estimators=50, random_state=0).fit(X, y)7 8explainer = shap.TreeExplainer(model)9sv = explainer(X.iloc[:200])10 11shap.plots.waterfall(sv[0])     # why house 0 got its price12shap.plots.beeswarm(sv)         # which features matter overall

1/4Train a random forest to predict house prices from eight features.

A waterfall plot explains one prediction. A beeswarm plot shows the whole picture across many rows.

Save yourself the pain

Common mistakes

Oops

Saying 'income causes lower default' because SHAP shows it lowers predicted risk.

Do this instead

SHAP explains the model's behaviour. Proving cause needs experiments or careful causal analysis.

Oops

Trusting one LIME run as the final answer.

Do this instead

LIME uses random samples, so results can shift between runs. Run it a few times and check the story holds.

Your mission

Explain two houses

~30 min

Quick check

The base value is 0.20 and the prediction is 0.50. Three features have SHAP values +0.25, -0.05 and x. What is x?

Go deeper

1Apply SHAP to a trained model and interpret the output for a single prediction.
2Learn the conceptual difference between global explainability (the whole model) and local (one prediction).
3Understand why regulated industries often require this kind of explainability by law.