Table of Contents

Data Drift and Concept Drift Detection

Rumman Ansari September 16, 2026 22 views Subject Details

MLOps Fundamentals

Data Drift and Concept Drift Detection

Learn how to detect and investigate changes in production data, predictions, labels, and feature-to-outcome relationships using reference datasets, monitoring windows, statistical tests, distance measures, delayed ground truth, segment analysis, alerts, and responsible retraining workflows.

Introduction

Machine Learning models learn patterns from historical data. After deployment, the real-world environment continues to change. Customer behavior, business policies, products, sensors, source systems, economic conditions, data pipelines, and operational processes may evolve.

These changes can cause production data to differ from the data used to train and evaluate the model. They can also change the relationship between model inputs and the outcome being predicted.

Data drift describes changes in input-data distributions. Concept drift describes changes in the relationship between inputs and outcomes. Both are important, but they require different evidence and may require different responses.

Core principle: Drift is a signal that the operating environment has changed. It is not automatic proof that the model has failed or must be retrained.

Learning Objectives

By the End of This Lesson

Define data drift and concept drift
Distinguish feature, prediction, label, and performance drift
Explain why drift occurs in production systems
Select a suitable reference dataset
Define current monitoring windows
Monitor numerical and categorical features
Calculate Population Stability Index
Apply the Kolmogorov-Smirnov test
Apply the chi-squared test to categorical data
Calculate Jensen-Shannon divergence
Understand the limitations of statistical significance
Detect prediction drift before labels arrive
Use delayed labels to investigate concept drift
Compare model performance across time windows
Detect segment-specific drift
Account for seasonality and recurring patterns
Design actionable drift alerts
Investigate drift systematically
Decide when retraining is appropriate
Create a drift-detection readiness checklist

Prerequisites

Recommended Knowledge

Probability distributions
Mean, variance, and quantiles
Hypothesis testing
Classification and regression metrics
pandas and NumPy
scikit-learn
Model monitoring
Experiment tracking
ML pipelines
Delayed ground-truth labels

Packages Used in the Examples

pandas
NumPy
SciPy
scikit-learn
Evidently

python -m pip install pandas numpy scipy scikit-learn evidently

Data requirement: Drift detection requires an approved reference dataset and an actual current dataset. Do not invent production observations or use sensitive records without authorization.

What Is Drift?

Drift is a measurable change in data, predictions, labels, model quality, or the relationships used by a Machine Learning system. Drift can occur suddenly, gradually, incrementally, or as a recurring pattern.

Drift Type	What Changes?	Ground Truth Required?
Feature Drift	Distribution of one or more model-input features	No
Prediction Drift	Distribution of model scores, classes, or forecasts	No
Label Drift	Distribution of observed outcomes	Yes
Concept Drift	Relationship between inputs and outcomes	Generally yes
Performance Drift	Observed model-quality metrics	Yes
Attribution Drift	Contribution of features to predictions	Depends on the analysis

What Is Data Drift?

Data drift occurs when the production distribution of input variables differs from the selected reference distribution.

Data Drift

\[ P_{\text{reference}}(X) \neq P_{\text{current}}(X) \]

Data drift can affect one feature, a subset of features, or the complete population. It may reflect a genuine population change, seasonality, an upstream defect, a changed data definition, or a new operating segment.

Possible Causes

Changes in customer behavior
New products or services
Seasonal demand
Changed source-system logic
New sensors or devices
Modified data-collection processes
Unexpected missing values
New categorical values
Policy or eligibility changes
Population expansion into new regions

What Is Concept Drift?

Concept drift occurs when the relationship between input variables and the target outcome changes.

Concept Drift

\[ P_{\text{reference}}(Y \mid X) \neq P_{\text{current}}(Y \mid X) \]

A model may receive statistically similar inputs while becoming less accurate because those inputs no longer imply the same outcomes. Detecting this condition generally requires verified labels, business outcomes, or another reliable performance signal.

Important distinction: Data drift examines changes in \(P(X)\). Concept drift examines changes in \(P(Y \mid X)\).

Data Drift vs Concept Drift

Area	Data Drift	Concept Drift
Main Change	Input-data distribution	Feature-to-outcome relationship
Can Occur Without Labels	Can be detected without labels	Usually cannot be confirmed without outcomes
Typical Evidence	Distribution tests and distance measures	Performance, calibration, residual, and outcome changes
Automatic Retraining	Not automatically justified	May justify candidate retraining after investigation
Possible Cause	Population, source, schema, or process change	Changed real-world behavior or target relationship

Drift Patterns

Pattern	Description	Example Investigation
Sudden	A distribution changes sharply	Check deployments, sources, policies, and schema changes
Gradual	The population changes progressively	Compare rolling windows and long-term trends
Incremental	Small changes accumulate over time	Compare current data with both recent and original baselines
Recurring	A previous distribution returns periodically	Compare equivalent seasonal or operational periods
Temporary	A short-lived event creates an unusual distribution	Check incidents, promotions, outages, and exceptional events

Choose a Reference Dataset

Drift is always measured relative to a benchmark. The baseline must be meaningful for the monitoring objective.

Reference Option	Advantage	Limitation
Training Data	Represents the population used to fit the model	May not represent accepted production behavior
Validation Data	Connects drift with model-selection evidence	May contain fewer observations
Launch Period	Represents early production behavior	Must first be confirmed as valid
Previous Window	Detects recent changes	Slow drift can become normalized
Seasonal Baseline	Accounts for recurring behavior	Requires enough historical evidence
Approved Business Baseline	Represents an explicitly accepted operating population	Requires governance and maintenance

Baseline principle: Version the reference dataset, schema, time period, selection reason, feature definitions, and associated model version.

Select Monitoring Windows

Window	Use
Tumbling Window	Analyzes consecutive, non-overlapping periods
Rolling Window	Analyzes a moving recent period
Expanding Window	Accumulates data from a fixed starting point
Event Window	Analyzes data around a deployment or business event
Outcome-Maturity Window	Waits until labels are sufficiently complete
Seasonal Window	Compares equivalent recurring periods

Window trade-off: Short windows detect changes quickly but may be noisy. Long windows provide stability but can delay detection or hide local problems.

Validate Data Before Drift Analysis

Pre-Drift Checks

Reference and current datasets are non-empty
Schema versions are compatible
Feature definitions are consistent
Data types are correct
Missing and infinite values are measured separately
Duplicate events are handled
Time windows are complete
Sampling rules are consistent
Prediction and model versions are identifiable
Known incidents are documented

Incorrect Sequence Calculate a distribution test before checking whether an upstream field has changed type or become mostly missing.

Correct Sequence Validate schema and data quality first, then calculate and interpret distribution changes.

Population Stability Index

Population Stability Index, or PSI, compares the proportions of observations falling into corresponding reference and current bins.

PSI

\[ \operatorname{PSI} = \sum_{i=1}^{k} \left( A_i-E_i \right) \ln \left( \frac{A_i}{E_i} \right) \]

Here, \(E_i\) is the reference proportion and \(A_i\) is the current proportion in bin \(i\).

import numpy as np


def calculate_psi(
    reference_values,
    current_values,
    number_of_bins=10,
    epsilon=1e-6
):
    reference_values = np.asarray(
        reference_values,
        dtype=float
    )

    current_values = np.asarray(
        current_values,
        dtype=float
    )

    reference_values = reference_values[
        np.isfinite(
            reference_values
        )
    ]

    current_values = current_values[
        np.isfinite(
            current_values
        )
    ]

    if (
        reference_values.size == 0
        or current_values.size == 0
    ):
        raise ValueError(
            "Both datasets require finite values."
        )

    bin_edges = np.unique(
        np.quantile(
            reference_values,
            np.linspace(
                0,
                1,
                number_of_bins + 1
            )
        )
    )

    if bin_edges.size < 2:
        return 0.0

    bin_edges[
        0
    ] = -np.inf

    bin_edges[
        -1
    ] = np.inf

    reference_counts, _ = np.histogram(
        reference_values,
        bins=bin_edges
    )

    current_counts, _ = np.histogram(
        current_values,
        bins=bin_edges
    )

    reference_proportions = (
        reference_counts
        /
        reference_counts.sum()
    )

    current_proportions = (
        current_counts
        /
        current_counts.sum()
    )

    reference_proportions = np.clip(
        reference_proportions,
        epsilon,
        None
    )

    current_proportions = np.clip(
        current_proportions,
        epsilon,
        None
    )

    return float(
        np.sum(
            (
                current_proportions
                -
                reference_proportions
            )
            *
            np.log(
                current_proportions
                /
                reference_proportions
            )
        )
    )

PSI caution: PSI is sensitive to bin definitions, sample size, missing-value treatment, and feature distribution. Validate project-specific thresholds instead of applying universal cutoffs blindly.

Kolmogorov-Smirnov Test

The two-sample Kolmogorov-Smirnov test compares the empirical cumulative distributions of two numerical samples.

KS Statistic

\[ D = \sup_x \left| F_{\text{reference}}(x) - F_{\text{current}}(x) \right| \]

import numpy as np
from scipy.stats import ks_2samp


def calculate_ks_test(
    reference_values,
    current_values
):
    reference_values = np.asarray(
        reference_values,
        dtype=float
    )

    current_values = np.asarray(
        current_values,
        dtype=float
    )

    reference_values = reference_values[
        np.isfinite(
            reference_values
        )
    ]

    current_values = current_values[
        np.isfinite(
            current_values
        )
    ]

    if (
        reference_values.size == 0
        or current_values.size == 0
    ):
        raise ValueError(
            "Both datasets require finite values."
        )

    result = ks_2samp(
        reference_values,
        current_values
    )

    return {
        "statistic": float(
            result.statistic
        ),
        "p_value": float(
            result.pvalue
        ),
        "reference_count": int(
            reference_values.size
        ),
        "current_count": int(
            current_values.size
        )
    }

Statistical caution: A small p-value indicates evidence against equal distributions under the test assumptions. It does not measure business importance or prove that model quality has degraded.

Chi-Squared Test for Categorical Drift

import pandas as pd
from scipy.stats import chi2_contingency


def calculate_categorical_drift(
    reference_values,
    current_values
):
    reference_series = (
        pd.Series(
            reference_values
        )
        .fillna(
            "__MISSING__"
        )
        .astype(
            str
        )
    )

    current_series = (
        pd.Series(
            current_values
        )
        .fillna(
            "__MISSING__"
        )
        .astype(
            str
        )
    )

    categories = sorted(
        set(
            reference_series.unique()
        )
        |
        set(
            current_series.unique()
        )
    )

    reference_counts = (
        reference_series
        .value_counts()
        .reindex(
            categories,
            fill_value=0
        )
    )

    current_counts = (
        current_series
        .value_counts()
        .reindex(
            categories,
            fill_value=0
        )
    )

    comparison_table = np.vstack(
        [
            reference_counts.values,
            current_counts.values
        ]
    )

    statistic, p_value, _, _ = (
        chi2_contingency(
            comparison_table
        )
    )

    return {
        "statistic": float(
            statistic
        ),
        "p_value": float(
            p_value
        ),
        "categories": categories
    }

Category review: Monitor unseen categories, disappearing categories, changed spelling, placeholder values, missing-value shares, and low-count categories separately.

Jensen-Shannon Divergence

Jensen-Shannon divergence measures the difference between two probability distributions using their average distribution.

Jensen-Shannon Divergence

\[ \operatorname{JS}(P,Q) = \frac{1}{2} \operatorname{KL}(P \parallel M) + \frac{1}{2} \operatorname{KL}(Q \parallel M) \]

\[ M = \frac{1}{2} \left( P+Q \right) \]

import numpy as np
from scipy.spatial.distance import jensenshannon


def calculate_js_divergence(
    reference_proportions,
    current_proportions,
    epsilon=1e-12
):
    reference_proportions = np.asarray(
        reference_proportions,
        dtype=float
    )

    current_proportions = np.asarray(
        current_proportions,
        dtype=float
    )

    if (
        reference_proportions.shape
        !=
        current_proportions.shape
    ):
        raise ValueError(
            "Both distributions must "
            "have the same shape."
        )

    reference_proportions = np.clip(
        reference_proportions,
        epsilon,
        None
    )

    current_proportions = np.clip(
        current_proportions,
        epsilon,
        None
    )

    reference_proportions /= (
        reference_proportions.sum()
    )

    current_proportions /= (
        current_proportions.sum()
    )

    distance = jensenshannon(
        reference_proportions,
        current_proportions,
        base=2
    )

    return {
        "js_distance": float(
            distance
        ),
        "js_divergence": float(
            distance ** 2
        )
    }

Compare Category Distributions

import pandas as pd


def compare_category_distributions(
    reference_values,
    current_values
):
    reference_distribution = (
        pd.Series(
            reference_values
        )
        .fillna(
            "__MISSING__"
        )
        .value_counts(
            normalize=True
        )
    )

    current_distribution = (
        pd.Series(
            current_values
        )
        .fillna(
            "__MISSING__"
        )
        .value_counts(
            normalize=True
        )
    )

    categories = sorted(
        set(
            reference_distribution.index
        )
        |
        set(
            current_distribution.index
        )
    )

    comparison = pd.DataFrame(
        {
            "category": categories,
            "reference_proportion": [
                float(
                    reference_distribution.get(
                        category,
                        0.0
                    )
                )
                for category in categories
            ],
            "current_proportion": [
                float(
                    current_distribution.get(
                        category,
                        0.0
                    )
                )
                for category in categories
            ]
        }
    )

    comparison[
        "absolute_change"
    ] = (
        comparison[
            "current_proportion"
        ]
        -
        comparison[
            "reference_proportion"
        ]
    ).abs()

    return comparison.sort_values(
        by="absolute_change",
        ascending=False
    )

Feature-Level Drift Report

def create_numerical_drift_report(
    reference_frame,
    current_frame,
    numerical_features
):
    drift_rows = []

    for feature_name in numerical_features:

        psi_value = calculate_psi(
            reference_frame[
                feature_name
            ],
            current_frame[
                feature_name
            ]
        )

        ks_result = calculate_ks_test(
            reference_frame[
                feature_name
            ],
            current_frame[
                feature_name
            ]
        )

        drift_rows.append(
            {
                "feature": feature_name,
                "psi": psi_value,
                "ks_statistic": (
                    ks_result[
                        "statistic"
                    ]
                ),
                "ks_p_value": (
                    ks_result[
                        "p_value"
                    ]
                ),
                "reference_count": (
                    ks_result[
                        "reference_count"
                    ]
                ),
                "current_count": (
                    ks_result[
                        "current_count"
                    ]
                )
            }
        )

    return pd.DataFrame(
        drift_rows
    )

Interpretation requirement: Include sample sizes, missing-value rates, effect sizes, model sensitivity, feature importance, and business context alongside drift statistics.

Prediction Drift

Prediction drift occurs when model-output distributions change. It can be monitored before verified ground-truth outcomes become available.

Prediction Signal	Question
Mean Score	Has average predicted risk changed?
Score Distribution	Are predictions moving toward extremes or the threshold?
Positive Prediction Rate	Has the percentage crossing the threshold changed?
Class Distribution	Are predicted classes appearing at different rates?
Abstention Rate	Is the model declining more decisions?
Uncertainty	Are predictions becoming less decisive?

import numpy as np


def summarize_prediction_distribution(
    probabilities,
    decision_threshold
):
    probabilities = np.asarray(
        probabilities,
        dtype=float
    )

    probabilities = probabilities[
        np.isfinite(
            probabilities
        )
    ]

    if probabilities.size == 0:

        raise ValueError(
            "Prediction probabilities are required."
        )

    return {
        "sample_count": int(
            probabilities.size
        ),
        "mean_probability": float(
            probabilities.mean()
        ),
        "median_probability": float(
            np.median(
                probabilities
            )
        ),
        "positive_prediction_rate": float(
            (
                probabilities
                >=
                decision_threshold
            ).mean()
        ),
        "near_threshold_rate": float(
            (
                np.abs(
                    probabilities
                    -
                    decision_threshold
                )
                <=
                0.05
            ).mean()
        )
    }

Delayed Ground Truth

Concept and performance drift cannot be assessed reliably until predicted cases are connected with sufficiently mature verified outcomes.

Delayed Evaluation

Prediction → Outcome Matures → Validated Join → Performance Analysis

Required Join Fields

Prediction identifier
Prediction timestamp
Model version
Score or predicted value
Decision-threshold version
Outcome timestamp
Verified outcome
Label-definition version
Outcome-maturity status

Join Predictions with Outcomes

import pandas as pd


def join_predictions_and_outcomes(
    predictions,
    outcomes
):
    required_prediction_columns = {
        "prediction_id",
        "prediction_time",
        "model_version",
        "probability"
    }

    required_outcome_columns = {
        "prediction_id",
        "outcome_time",
        "observed_label"
    }

    if not required_prediction_columns.issubset(
        predictions.columns
    ):
        raise ValueError(
            "Prediction data is missing "
            "required columns."
        )

    if not required_outcome_columns.issubset(
        outcomes.columns
    ):
        raise ValueError(
            "Outcome data is missing "
            "required columns."
        )

    joined = predictions.merge(
        outcomes,
        on="prediction_id",
        how="left",
        validate="one_to_one"
    )

    joined[
        "label_available"
    ] = joined[
        "observed_label"
    ].notna()

    return joined

Join integrity: Validate uniqueness, timestamp order, outcome definition, duplicates, missing labels, and eligibility before calculating performance.

Detect Performance Drift

from sklearn.metrics import f1_score
from sklearn.metrics import precision_score
from sklearn.metrics import recall_score
from sklearn.metrics import roc_auc_score


def calculate_window_metrics(
    labelled_frame,
    decision_threshold
):
    labelled_frame = labelled_frame.dropna(
        subset=[
            "observed_label",
            "probability"
        ]
    )

    if labelled_frame.empty:

        raise ValueError(
            "Mature labelled observations "
            "are required."
        )

    true_labels = labelled_frame[
        "observed_label"
    ].astype(
        int
    )

    probabilities = labelled_frame[
        "probability"
    ].astype(
        float
    )

    predicted_labels = (
        probabilities
        >=
        decision_threshold
    ).astype(
        int
    )

    return {
        "sample_count": int(
            len(
                labelled_frame
            )
        ),
        "precision": float(
            precision_score(
                true_labels,
                predicted_labels,
                zero_division=0
            )
        ),
        "recall": float(
            recall_score(
                true_labels,
                predicted_labels,
                zero_division=0
            )
        ),
        "f1": float(
            f1_score(
                true_labels,
                predicted_labels,
                zero_division=0
            )
        ),
        "roc_auc": float(
            roc_auc_score(
                true_labels,
                probabilities
            )
        )
    }

Completeness requirement: Report sample size and label-completeness rate with every delayed quality result.

Calibration Drift

Calibration drift occurs when predicted probabilities no longer correspond well to observed frequencies.

Calibration

\[ P \left( Y=1 \mid \hat{p}=p \right) \approx p \]

from sklearn.metrics import brier_score_loss


def calculate_calibration_metric(
    true_labels,
    probabilities
):
    return {
        "brier_score": float(
            brier_score_loss(
                true_labels,
                probabilities
            )
        )
    }

Calibration importance: Ranking performance can remain stable while probability estimates become unreliable for threshold-based decisions.

Segment-Specific Drift

Aggregate analysis can hide changes within a region, product, device, acquisition channel, customer type, or another relevant operating segment.

def calculate_segment_psi(
    reference_frame,
    current_frame,
    segment_column,
    feature_name
):
    segments = sorted(
        set(
            reference_frame[
                segment_column
            ].dropna()
        )
        |
        set(
            current_frame[
                segment_column
            ].dropna()
        )
    )

    rows = []

    for segment in segments:

        reference_values = (
            reference_frame.loc[
                reference_frame[
                    segment_column
                ]
                ==
                segment,
                feature_name
            ]
        )

        current_values = (
            current_frame.loc[
                current_frame[
                    segment_column
                ]
                ==
                segment,
                feature_name
            ]
        )

        if (
            reference_values.empty
            or current_values.empty
        ):
            rows.append(
                {
                    "segment": segment,
                    "psi": None,
                    "status": "insufficient-data"
                }
            )

            continue

        rows.append(
            {
                "segment": segment,
                "psi": calculate_psi(
                    reference_values,
                    current_values
                ),
                "status": "calculated"
            }
        )

    return pd.DataFrame(
        rows
    )

Group protection: Apply minimum sample-size rules and privacy controls before reporting small-group results.

Seasonality and Expected Change

Not every distribution change is harmful. Some changes are expected because of recurring business cycles, seasons, weekends, campaigns, maintenance periods, or operational schedules.

Seasonal Review

Compare equivalent periods
Document recurring business events
Use time-aware features and windows
Review whether seasonality was present during training
Separate expected change from unexpected change
Monitor model quality during seasonal peaks
Do not suppress alerts without documented evidence

Multivariate Drift

Univariate methods examine one feature at a time. They may miss changes in relationships among multiple features.

Approach	Purpose	Consideration
Domain Classifier	Predicts whether a record came from reference or current data	Requires careful validation and interpretation
Embedding Distance	Compares vector representations	Depends on the embedding model and distance measure
Maximum Mean Discrepancy	Compares multivariate distributions through kernels	Requires kernel and threshold selection
Principal Component Analysis	Projects data to inspect broad population changes	Can hide changes outside leading components
Cluster Comparison	Examines changes in discovered population structure	Depends on clustering stability

Layered approach: Use simple feature-level methods for explainability and add multivariate methods when relationships among features are important.

Domain-Classifier Drift Detection

A domain classifier labels reference records as one class and current records as another. If a classifier separates the two groups consistently, their multivariate distributions may differ.

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split


def calculate_domain_classifier_auc(
    reference_frame,
    current_frame,
    random_state=42
):
    reference_copy = reference_frame.copy()

    current_copy = current_frame.copy()

    reference_copy[
        "__domain__"
    ] = 0

    current_copy[
        "__domain__"
    ] = 1

    combined = pd.concat(
        [
            reference_copy,
            current_copy
        ],
        ignore_index=True
    )

    features = combined.drop(
        columns=[
            "__domain__"
        ]
    )

    target = combined[
        "__domain__"
    ]

    (
        training_features,
        test_features,
        training_labels,
        test_labels
    ) = train_test_split(
        features,
        target,
        test_size=0.3,
        random_state=random_state,
        stratify=target
    )

    classifier = LogisticRegression(
        max_iter=2000,
        random_state=random_state
    )

    classifier.fit(
        training_features,
        training_labels
    )

    probabilities = classifier.predict_proba(
        test_features
    )[:, 1]

    return {
        "domain_auc": float(
            roc_auc_score(
                test_labels,
                probabilities
            )
        )
    }

Interpretation caution: A domain classifier identifies separability, not business harm. Investigate which features drive the distinction and whether they affect model quality.

Drift Detection with Evidently

from evidently import Report
from evidently.presets import DataDriftPreset


drift_report = Report(
    [
        DataDriftPreset()
    ]
)


drift_result = drift_report.run(
    current_data=current_frame,
    reference_data=reference_frame
)


drift_result.save_html(
    "reports/data_drift_report.html"
)

Library compatibility: APIs can change between package versions. Use the documentation matching the tested Evidently version in your environment.

Run Drift Tests with Evidently

from evidently import Report
from evidently.presets import DataDriftPreset


drift_test_report = Report(
    [
        DataDriftPreset(
            include_tests=True
        )
    ]
)


test_result = drift_test_report.run(
    current_data=current_frame,
    reference_data=reference_frame
)


test_result.save_html(
    "reports/data_drift_tests.html"
)

Test configuration: Review the methods, defaults, thresholds, feature types, and sample sizes used by the installed library. Configure project-specific behavior where required.

Build a Layered Detection Strategy

Validate Schema and Quality

Check missing values, types, categories, ranges, duplicates, timestamps, and source completeness.

Measure Feature Drift

Compare numerical and categorical feature distributions against approved baselines.

Measure Prediction Drift

Track changes in scores, classes, selection rates, uncertainty, and abstention.

Analyze Important Segments

Identify drift hidden within products, regions, devices, channels, or relevant operating groups.

Join Delayed Outcomes

Connect predictions to sufficiently mature verified labels.

Evaluate Performance and Calibration

Measure whether model quality and probability reliability have changed.

Investigate Business Impact

Review outcomes, overrides, complaints, fallback usage, and operational value.

Choose a Controlled Response

Continue monitoring, correct data, adjust policy, restrict use, evaluate a candidate, roll back, or retire the model.

Design Drift Alerts

Every Alert Should Include

Model and application version
Feature, prediction, or quality metric
Reference dataset
Current window
Detection method
Observed value
Threshold
Sample size
Missing-value rate
Severity
Responsible owner
Investigation procedure
Escalation and mitigation rules

Weak Alert Data drift detected.

Actionable Alert Identify the affected feature or segment, model version, baseline, current window, method, observed value, sample size, threshold, owner, and next investigation step.

Reduce False Alerts

Alert-Quality Controls

Require a meaningful minimum sample size
Require persistence across multiple windows where appropriate
Combine statistical significance with effect size
Account for approved seasonal patterns
Separate missing-value alerts from distribution tests
Prioritize model-sensitive features
Group correlated alerts
Use warning and critical severity levels
Review thresholds after incidents and false positives
Retire alerts that do not lead to useful action

Drift Investigation Workflow

Verify the Calculation

Confirm the baseline, current window, sample size, missing-value handling, feature type, method, and threshold.

Check Data Quality

Review schema, source systems, transformations, missing values, duplicates, timestamp logic, and category mappings.

Identify the Scope

Determine the affected model versions, periods, features, segments, endpoints, and business processes.

Inspect Distribution Changes

Compare summary statistics, quantiles, categories, plots, and related features.

Assess Model Sensitivity

Determine whether changed features are used strongly by the model or affect important decision regions.

Review Outcomes

Use mature labels to evaluate performance, calibration, groups, and business impact.

Mitigate

Correct data, restrict use, apply fallback, roll back, or pause the workflow where required.

Document and Improve

Record root cause, impact, response, recovery evidence, and monitoring changes.

Should Drift Trigger Retraining?

Observed Condition	Possible Response
Schema or Pipeline Defect	Correct the upstream system or transformation
Expected Seasonal Change	Continue monitoring with an appropriate seasonal baseline
Data Drift Without Quality Loss	Investigate and continue controlled observation
Prediction Drift Without Labels	Prioritize outcome collection and model review
Confirmed Performance Degradation	Investigate data, model, threshold, and concept causes
Confirmed Concept Change	Train and independently evaluate a candidate
Changed Business Objective	Redefine the problem before retraining
Policy Restriction	Restrict or retire the model regardless of technical performance

Unsafe Automation Detect drift and immediately replace the deployed model.

Controlled Automation Validate the signal, investigate its cause, create a traceable candidate, compare it with the approved model, and release only after required gates pass.

Rollback and Fallback

Fallback

Provides alternate behavior during uncertainty or failure
May use human review, rules, queues, or safe defaults
Must be monitored and tested

Rollback

Restores a previous approved release
Requires compatible preprocessing, schema, and threshold
Requires post-restoration verification

Scheduled Drift Pipeline

Drift-Detection Pipeline

Load Reference → Load Current Window → Validate → Calculate Drift → Analyze Segments → Publish Report → Evaluate Alerts

Pipeline Metadata

Pipeline-run identifier
Model version
Reference-data version
Current-window boundaries
Feature-definition version
Detection-method version
Threshold configuration
Sample counts
Report location
Alert result

Create a Drift Report

import json
from pathlib import Path


def save_drift_report(
    model_version,
    reference_version,
    current_window,
    numerical_results,
    categorical_results,
    prediction_results,
    output_path
):
    report = {
        "model_version": model_version,
        "reference_version": (
            reference_version
        ),
        "current_window": current_window,
        "numerical_drift": (
            numerical_results
        ),
        "categorical_drift": (
            categorical_results
        ),
        "prediction_drift": (
            prediction_results
        )
    }

    output_path = Path(
        output_path
    )

    output_path.parent.mkdir(
        parents=True,
        exist_ok=True
    )

    with open(
        output_path,
        "w",
        encoding="utf-8"
    ) as report_file:

        json.dump(
            report,
            report_file,
            indent=4
        )

    return report

Report requirement: Preserve the baseline, monitoring window, sample sizes, model version, calculation version, missing-value treatment, and actual measured results.

Test Drift-Detection Logic

import numpy as np
import pytest


def test_psi_is_zero_for_identical_data():

    values = np.array(
        [
            1,
            2,
            3,
            4,
            5
        ],
        dtype=float
    )

    result = calculate_psi(
        values,
        values,
        number_of_bins=5
    )

    assert result == pytest.approx(
        0.0
    )


def test_psi_rejects_empty_data():

    with pytest.raises(
        ValueError,
        match="finite values"
    ):
        calculate_psi(
            [],
            [
                1,
                2,
                3
            ]
        )


def test_prediction_summary_range():

    summary = (
        summarize_prediction_distribution(
            probabilities=np.array(
                [
                    0.1,
                    0.4,
                    0.7,
                    0.9
                ]
            ),
            decision_threshold=0.6
        )
    )

    assert (
        0.0
        <=
        summary[
            "positive_prediction_rate"
        ]
        <=
        1.0
    )

python -m pytest -v

Security and Privacy

Required Controls

Use only authorized monitoring data
Minimize retained input values
Protect personal and confidential information
Use controlled identifiers
Restrict access to reference and current datasets
Encrypt stored reports
Protect exported plots and tables
Define retention and deletion rules
Audit access and downloads
Prevent small-group re-identification

Privacy principle: Drift detection usually requires distributions and aggregates. Do not retain complete identifiable records when approved aggregate monitoring can satisfy the objective.

Responsible Drift Detection

Responsible-Use Requirements

Document intended and prohibited model uses
Monitor relevant operating groups
Use minimum sample-size controls
Communicate uncertainty in drift conclusions
Do not confuse correlation with cause
Investigate unequal performance changes
Include qualified human review
Track user complaints and overrides
Preserve drift and incident evidence
Maintain fallback and rollback
Retire models that are no longer suitable

Common Drift-Detection Mistakes

Mistake Treat every statistical difference as a production incident.

Correction Consider effect size, sample size, model sensitivity, seasonality, performance, and business impact.

Mistake Use one universal threshold for all features.

Correction Validate thresholds according to the feature distribution, method, operating risk, and model reliance.

Mistake Ignore missing-value drift.

Correction Monitor null and invalid-value rates separately from non-null distribution tests.

Mistake Use the previous window as the only baseline.

Correction Compare against recent, launch, training, and seasonal baselines as appropriate.

Mistake Infer concept drift from feature drift alone.

Correction Use mature outcomes to evaluate changing feature-to-label relationships and model performance.

Mistake Retrain automatically whenever drift is detected.

Correction Investigate the cause, validate new data, evaluate a candidate, and apply approval gates.

Mistake Monitor only aggregate drift.

Correction Analyze relevant features, segments, predictions, outcomes, and business processes.

Recommended Drift-Detection Workflow

Define the Monitoring Objective

Identify the model, outcome, risks, important features, segments, owners, and expected response.

Select and Version Baselines

Preserve suitable reference data, schema, feature definitions, time periods, and selection reasons.

Define Monitoring Windows

Choose window length, frequency, completeness, and seasonal comparisons.

Validate Data Quality

Detect schema, missing-value, category, range, duplicate, and source problems before drift analysis.

Calculate Feature and Prediction Drift

Use suitable statistical and distance methods for each feature type.

Analyze Segments

Identify changes hidden inside regional, product, channel, device, or other relevant groups.

Join Outcomes

Connect predictions to mature verified labels and calculate performance and calibration.

Investigate Alerts

Determine whether the cause is data quality, natural change, concept change, model behavior, or business policy.

Apply a Controlled Response

Correct data, continue monitoring, evaluate a candidate, use fallback, roll back, or retire the model.

Improve Detection

Review false alerts, missed incidents, thresholds, windows, baselines, privacy, cost, and owner response.

Hands-On Practice

Create Reference and Current Datasets

Use approved historical and recent datasets with compatible schemas.

Validate Data Quality

Compare missing values, categories, ranges, duplicates, and schema versions.

Calculate Numerical Drift

Apply PSI and the KS test to selected numerical features.

Calculate Categorical Drift

Compare category shares and apply a suitable statistical test.

Measure Prediction Drift

Compare score distributions, positive rates, and observations near the decision threshold.

Join Ground Truth

Connect historical predictions with mature outcomes and calculate quality metrics.

Create an Evidently Report

Generate a visual drift report and inspect the selected methods, samples, distributions, and results.

Design an Incident Response

Define the alert, owner, investigation, fallback, candidate evaluation, and recovery verification.

Knowledge Check

What is data drift?

Explain changes in production input distributions relative to an identified reference.

What is concept drift?

Explain changes in the relationship between model inputs and target outcomes.

What is prediction drift?

Explain changes in scores, predicted classes, forecasts, or threshold-selection rates.

Why is the baseline important?

Explain that every drift result is relative to an identified reference distribution.

What does PSI compare?

Explain reference and current proportions across corresponding bins.

What does the KS statistic measure?

Explain the largest difference between two empirical cumulative distributions.

Why can statistical significance be misleading?

Explain sample-size sensitivity and the difference between statistical and practical importance.

Why are delayed labels required?

Explain that concept and performance drift require verified outcomes.

Why should drift not automatically trigger retraining?

Explain validation, diagnosis, candidate evaluation, approval, and controlled promotion.

Why is segment-level drift analysis important?

Explain how aggregate stability can hide meaningful changes within particular operating groups.

Drift-Detection Readiness Checklist

Confirm Before Production Use

The monitored model and application versions are identifiable
The drift-monitoring objective is documented
Reference datasets are approved and versioned
Current monitoring windows are defined
Seasonal comparison rules are defined
Reference and current schemas are validated
Missing values are monitored separately
Numerical feature methods are selected appropriately
Categorical feature methods are selected appropriately
Prediction drift is monitored
Label completeness is measured
Delayed ground-truth joining is validated
Performance drift is monitored
Calibration is monitored where required
Relevant segments are monitored
Minimum sample sizes are defined
Statistical and practical importance are considered
Thresholds are project-specific
Alert persistence rules are defined
Every important alert has an owner
Investigation procedures are documented
False-alert review is scheduled
Drift logic is versioned and tested
Reports include model, baseline, window, method, and samples
Monitoring data is minimized and protected
Retraining produces a candidate only
Fallback and rollback are tested
Responsible-use requirements are documented
Model-retirement criteria are defined

Lesson Summary

Drift Area	Main Outcome
Data Drift	Detects changes in feature distributions
Prediction Drift	Detects changes in model-output distributions
Label Drift	Detects changes in observed outcome distributions
Concept Drift	Detects changes in feature-to-outcome relationships
Performance Drift	Detects degradation in measured model quality
Reference Dataset	Provides the benchmark for comparison
Monitoring Window	Defines the current population being evaluated
PSI	Compares reference and current bin proportions
KS Test	Compares empirical numerical distributions
Chi-Squared Test	Compares categorical frequency distributions
Jensen-Shannon Divergence	Measures difference between probability distributions
Delayed Ground Truth	Enables performance and concept-drift evaluation
Segment Analysis	Reveals drift hidden by aggregate measures
Alerting	Directs owners to validated investigation steps
Controlled Retraining	Creates an evaluated candidate after justified need

Key Takeaway

Data drift and concept drift detection help teams understand whether a deployed Machine Learning system continues to operate under familiar and valid conditions. Data drift measures changes in input distributions, while concept drift concerns changes in the relationship between inputs and outcomes. Feature and prediction drift can be monitored before labels arrive, but verified outcomes are generally required to assess performance, calibration, and concept drift. Reliable detection combines versioned baselines, appropriate windows, data-quality checks, statistical tests, distance measures, segment analysis, delayed labels, and business evidence. Drift should trigger investigation, not automatic replacement. Any retrained model remains a candidate until it passes independent evaluation, security, governance, responsible-use, deployment, and monitoring gates.

Get Machine Learning updates by email

Short, practical lessons and course updates from Rumman Ansari. No spam.