Introduction

Offline evaluation tells us how a model or feature performed on historical data. It does not always tell us what will happen when real users interact with the changed system.

A/B testing is a controlled experiment in which eligible experimental units are randomly assigned to different versions of an experience. Their outcomes are then compared using predefined metrics.

Version A is commonly the current experience, called the control. Version B is the proposed change, called the treatment or variant. Random assignment helps make the groups comparable before the treatment is applied.

Learning Objectives

Prerequisites

python -m pip install numpy pandas scipy statsmodels

Control and Treatment

Correlation vs Causation

Observational data can reveal that two variables move together, but this does not prove that one caused the other. Users who choose a feature may already differ from users who do not choose it.

Random assignment helps break systematic relationships between treatment assignment and pre-existing user characteristics. If the experiment is implemented correctly, differences in outcomes can be attributed more credibly to the treatment.

Choose the Experimental Unit

The experimental unit is the entity independently assigned to a variant. It should match how the treatment operates and how interference can occur.

Random Assignment

Random assignment gives eligible experimental units a defined chance of receiving each variant. It is the foundation of a controlled experiment.

Write the Hypothesis First

A useful hypothesis describes the change, affected population, expected mechanism, primary outcome, expected direction, and decision the result will support.

Define Metrics Before Launch

Conversion Rate

A binary conversion metric records whether an experimental unit completed a defined outcome.

If the unit is a user, one user should not be counted as several independent observations merely because the user generated several events.

Absolute and Relative Change

Report both absolute and relative changes. A relative percentage without the baseline can be difficult to interpret.

Null and Alternative Hypotheses

The null hypothesis represents no difference between variants. The alternative represents a difference. A directional alternative should be justified and chosen before observing the result.

Type I and Type II Errors

Understand the P-Value

A p-value measures how surprising the observed result, or a more extreme result, would be if the null hypothesis and statistical assumptions were true.

Confidence Intervals

A confidence interval communicates the estimated effect together with uncertainty. It is usually more informative than a significance label alone.

Statistical vs Practical Significance

A statistically detectable result may still be too small to justify implementation. The treatment may require engineering work, create support costs, increase latency, or introduce operational risk.

Minimum Detectable Effect

The minimum detectable effect, commonly abbreviated as MDE, is the effect size the experiment is designed to detect with the selected significance level and statistical power.

Plan Sample Size in Python

The following example demonstrates sample-size planning for two independent conversion proportions. The values are illustrative rather than universal recommendations.

from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize


baseline_rate = 0.10
minimum_target_rate = 0.11

effect_size = proportion_effectsize(
    baseline_rate,
    minimum_target_rate
)

power_analysis = NormalIndPower()

sample_size_per_group = power_analysis.solve_power(
    effect_size=effect_size,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative="two-sided"
)

print(
    round(sample_size_per_group)
)

Analyze a Binary Conversion Experiment

This example compares two independent conversion proportions using aggregated synthetic results.

import numpy as np

from statsmodels.stats.proportion import proportions_ztest
from statsmodels.stats.proportion import confint_proportions_2indep


control_conversions = 1020
control_users = 10000

treatment_conversions = 1110
treatment_users = 10000


control_rate = (
    control_conversions
    / control_users
)

treatment_rate = (
    treatment_conversions
    / treatment_users
)

absolute_change = (
    treatment_rate
    - control_rate
)

relative_change = (
    absolute_change
    / control_rate
)


conversion_counts = np.array([
    control_conversions,
    treatment_conversions
])

sample_sizes = np.array([
    control_users,
    treatment_users
])


test_statistic, p_value = proportions_ztest(
    count=conversion_counts,
    nobs=sample_sizes,
    alternative="two-sided"
)


confidence_interval = confint_proportions_2indep(
    count1=treatment_conversions,
    nobs1=treatment_users,
    count2=control_conversions,
    nobs2=control_users,
    method="wald",
    compare="diff",
    alpha=0.05
)


print("Control rate:", control_rate)
print("Treatment rate:", treatment_rate)
print("Absolute change:", absolute_change)
print("Relative change:", relative_change)
print("Test statistic:", test_statistic)
print("P-value:", p_value)
print("95% confidence interval:", confidence_interval)

Analyze a Continuous Metric

Revenue, order value, session duration, and latency are examples of continuous metrics. Their distributions may be skewed or contain extreme values, so the analysis plan should reflect the metric's behavior.

import numpy as np

from scipy.stats import ttest_ind


rng = np.random.default_rng(42)

control_revenue = rng.gamma(
    shape=2.0,
    scale=20.0,
    size=5000
)

treatment_revenue = rng.gamma(
    shape=2.05,
    scale=20.0,
    size=5000
)


test_statistic, p_value = ttest_ind(
    treatment_revenue,
    control_revenue,
    equal_var=False
)


mean_difference = (
    treatment_revenue.mean()
    - control_revenue.mean()
)


print(
    "Control mean:",
    control_revenue.mean()
)

print(
    "Treatment mean:",
    treatment_revenue.mean()
)

print(
    "Mean difference:",
    mean_difference
)

print(
    "P-value:",
    p_value
)

Run an A/A Test

An A/A test sends both groups to the same experience while retaining the experiment assignment and measurement pipeline.

Sample-Ratio Mismatch

Sample-ratio mismatch occurs when observed variant counts differ unexpectedly from the planned allocation beyond plausible random variation.

Repeated Peeking and Early Stopping

Repeatedly checking a conventional fixed-horizon test and stopping as soon as significance appears can increase false-positive risk.

Multiple Testing

Testing many metrics, variants, subgroups, and hypotheses increases the chance that at least one apparently positive result occurs by chance.

Novelty and Carryover Effects

Interference and Network Effects

Standard analysis often assumes that one unit's treatment does not change another unit's outcome. This assumption can fail in social networks, marketplaces, collaborative tools, games, and shared accounts.

A/B Testing Machine Learning Systems

A model can improve offline accuracy but fail to improve the product outcome. It may increase latency, reduce diversity, change user behavior, or optimize a proxy that does not represent long-term value.

Recommended Experiment Workflow

Common Experimentation Mistakes

Experiment Decision Record

Hands-On Practice

Knowledge Check

Lesson Summary