Introduction

A Machine Learning model can achieve strong evaluation results while remaining difficult to understand. A performance metric may show how accurately the model predicts, but it does not explain which inputs influenced the predictions or whether the model learned reasonable patterns.

Model interpretability helps people investigate how a model behaves. It can support debugging, feature review, stakeholder communication, risk analysis, fairness assessment, and human oversight.

SHAP and LIME are widely used post-hoc explanation methods. SHAP attributes a model output to input features using ideas from Shapley values. LIME creates an interpretable local surrogate model around one selected prediction.

Learning Objectives

Prerequisites

python -m pip install numpy pandas scikit-learn shap lime matplotlib

Why Interpretability Matters

Interpretability vs Explainability

The terms interpretability and explainability are often used interchangeably. A useful practical distinction is to treat interpretability as direct understanding of model mechanics and explainability as a human-understandable description of behavior, sometimes generated after the model has been trained.

Intrinsic and Post-Hoc Interpretability

Global vs Local Explanations

Model-Specific vs Model-Agnostic Methods

Built-In Feature Importance

Some estimators expose built-in feature-importance values. Tree models may report split-based importance, while linear models expose coefficients.

Permutation Importance

Permutation importance measures how much model performance changes when one feature is randomly shuffled. Shuffling breaks the relationship between that feature and the target while leaving the other columns unchanged.

In this expression, \(S_{\text{baseline}}\) is the original evaluation score and \(S_{\text{permuted }j}\) is the score after shuffling feature \(j\).

import pandas as pd

from sklearn.inspection import permutation_importance


permutation_result = permutation_importance(
    estimator=trained_pipeline,
    X=X_test,
    y=y_test,
    scoring="roc_auc",
    n_repeats=20,
    random_state=42,
    n_jobs=-1
)


importance_table = pd.DataFrame({
    "feature": X_test.columns,
    "mean_importance": permutation_result.importances_mean,
    "importance_std": permutation_result.importances_std
}).sort_values(
    by="mean_importance",
    ascending=False
)


print(
    importance_table.head(15)
)

What Is SHAP?

SHAP stands for SHapley Additive exPlanations. It uses ideas from cooperative game theory to assign feature contributions to a model output.

The prediction is represented as a baseline value plus feature-level contributions.

Shapley Value Intuition

Imagine that the input features are participants cooperating to produce a prediction. A feature's Shapley value is based on its average marginal contribution across possible feature coalitions.

The formula considers how much feature \(j\) changes the model output when added to different subsets of other features.

Prepare a Classification Model

The following example trains a Random Forest classifier on a built-in scikit-learn dataset. The dataset is used only to demonstrate the interpretation workflow.

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split


dataset = load_breast_cancer(
    as_frame=True
)

X = dataset.data
y = dataset.target


X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y
)


model = RandomForestClassifier(
    n_estimators=300,
    random_state=42,
    n_jobs=-1
)


model.fit(
    X_train,
    y_train
)


print(
    "Test accuracy:",
    model.score(
        X_test,
        y_test
    )
)

Generate SHAP Values

import shap


background_data = X_train.sample(
    n=min(200, len(X_train)),
    random_state=42
)


explainer = shap.Explainer(
    model,
    background_data
)


X_explain = X_test.iloc[:100].copy()


shap_values = explainer(
    X_explain
)


print(
    "SHAP value shape:",
    shap_values.values.shape
)

The explainer choice may depend on the estimator. Modern SHAP APIs can select an appropriate explainer automatically in many standard cases, but the resulting output dimensions must still be inspected.

SHAP Global Importance

Mean absolute SHAP values summarize the average magnitude of each feature's contributions across the explained observations.

import numpy as np
import pandas as pd


values = shap_values.values


if values.ndim == 3:
    positive_class_values = values[:, :, 1]
else:
    positive_class_values = values


global_shap_importance = pd.DataFrame({
    "feature": X_explain.columns,
    "mean_absolute_shap": np.abs(
        positive_class_values
    ).mean(axis=0)
}).sort_values(
    by="mean_absolute_shap",
    ascending=False
)


print(
    global_shap_importance.head(15)
)

SHAP Beeswarm Plot

A beeswarm plot combines global importance, contribution direction, feature value, and observation-level variation.

classification_values = shap_values


if shap_values.values.ndim == 3:
    classification_values = shap_values[:, :, 1]


shap.plots.beeswarm(
    classification_values,
    max_display=15
)

SHAP Waterfall Plot

A waterfall plot explains one prediction by starting from the baseline and showing how individual feature contributions move the output toward the final prediction.

observation_index = 0


local_explanation = classification_values[
    observation_index
]


shap.plots.waterfall(
    local_explanation,
    max_display=15
)

SHAP Dependence and Scatter Plots

A SHAP scatter or dependence plot shows how one feature's value relates to its attributed contribution across observations.

feature_name = "mean radius"


shap.plots.scatter(
    classification_values[
        :,
        feature_name
    ]
)

SHAP Additivity

A useful property of additive explanations is that the feature contributions combine with the baseline to reproduce the explained model output, subject to the explainer's output scale and numerical precision.

import numpy as np


local_values = local_explanation.values
base_value = local_explanation.base_values


reconstructed_output = (
    base_value
    + np.sum(local_values)
)


print(
    "Reconstructed explained output:",
    reconstructed_output
)

What Is LIME?

LIME stands for Local Interpretable Model-agnostic Explanations. It explains one prediction by generating perturbed examples near that observation, obtaining predictions from the original model, and fitting a simpler local surrogate model.

LIME Classification Example

import numpy as np

from lime.lime_tabular import LimeTabularExplainer


lime_explainer = LimeTabularExplainer(
    training_data=X_train.to_numpy(),
    feature_names=X_train.columns.tolist(),
    class_names=dataset.target_names.tolist(),
    mode="classification",
    discretize_continuous=True,
    random_state=42
)


observation_index = 0

observation = X_test.iloc[
    observation_index
].to_numpy()


lime_explanation = lime_explainer.explain_instance(
    data_row=observation,
    predict_fn=model.predict_proba,
    num_features=10
)


for feature_rule, contribution in lime_explanation.as_list():
    print(
        feature_rule,
        contribution
    )

The returned rules describe the locally fitted surrogate rather than the complete internal logic of the original model.

Export a LIME Explanation

lime_explanation.save_to_file(
    "lime_local_explanation.html"
)

LIME can produce an HTML explanation for review. Before sharing it, verify that it does not expose confidential feature values, personal information, hidden business rules, or sensitive model details.

SHAP vs LIME

Correlated Features

Correlated features create attribution challenges because several variables may contain overlapping information. An explanation method must allocate credit among features that can substitute for one another.

Explain the Complete Pipeline

Production models often include imputation, scaling, category encoding, feature generation, selection, and the estimator. Explanations must account for the representation actually received by the model.

Explain a Mixed-Feature Pipeline

The following pattern shows how transformed feature names can be retrieved from a fitted scikit-learn preprocessing pipeline.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import StandardScaler


numeric_pipeline = Pipeline(
    steps=[
        (
            "imputer",
            SimpleImputer(
                strategy="median"
            )
        ),
        (
            "scaler",
            StandardScaler()
        )
    ]
)


categorical_pipeline = Pipeline(
    steps=[
        (
            "imputer",
            SimpleImputer(
                strategy="most_frequent"
            )
        ),
        (
            "encoder",
            OneHotEncoder(
                handle_unknown="ignore"
            )
        )
    ]
)


preprocessor = ColumnTransformer(
    transformers=[
        (
            "numeric",
            numeric_pipeline,
            numeric_columns
        ),
        (
            "categorical",
            categorical_pipeline,
            categorical_columns
        )
    ]
)


model_pipeline = Pipeline(
    steps=[
        (
            "preprocessor",
            preprocessor
        ),
        (
            "model",
            model
        )
    ]
)


model_pipeline.fit(
    X_train,
    y_train
)


transformed_feature_names = (
    model_pipeline
    .named_steps["preprocessor"]
    .get_feature_names_out()
)


print(
    transformed_feature_names
)

Classification Output Scales

For classification, an explanation may describe probability, raw decision score, log odds, or another estimator-specific output. These values are not interchangeable.

Regression Explanations

For a regression model, SHAP values commonly describe how features move the numerical prediction relative to a baseline prediction.

Explanations Are Not Causal Effects

SHAP and LIME describe the behavior of a fitted predictive model. They do not establish what would happen if a person or organization deliberately changed a feature.

Explanation Stability

A reliable explanation should not change unpredictably when the random seed, background sample, perturbation sample, or a tiny input change is modified without a meaningful reason.

Explanation Fidelity

Fidelity describes how accurately an explanation represents the behavior of the original model in the relevant region.

Interpretability and Fairness

Explanations can help identify sensitive attributes, proxy features, and unusual subgroup patterns. However, feature attributions alone do not prove that a model is fair.

Privacy and Security

Explanation artifacts can reveal input values, feature names, business logic, model behavior, or sensitive information. They should be protected according to the same security and privacy requirements that apply to model inputs and predictions.

Explain Results for the Audience

Common Interpretation Mistakes

Recommended Interpretability Workflow

Explanation Record

Hands-On Practice

Knowledge Check

Lesson Summary