1. A Three-Layer Debugging Model

Scientific programs often fail for reasons that are more fundamental than the final algorithm. Before debugging a model's accuracy, verify that the computation is structurally valid, numerically valid and experimentally reproducible.

2. Prerequisites

python -m pip install numpy pandas torch

3. Diagnose Shape Errors by Writing Contracts

Do not begin with trial-and-error reshaping. First write the intended meaning of each axis. A shape is useful only when its axes have semantics.

For a common supervised-learning design:

where:

• \(B\) = batch size
• \(F\) = number of input features
• \(H\) = number of output or hidden features

The intended linear transformation is:

4. Derive the Expected Shape Before Running Code

Matrix multiplication contracts the inner dimension:

The bias then broadcasts across the batch:

Therefore:

import numpy as np

B, F, H = 32, 10, 4

X = np.ones((B, F))
W = np.ones((F, H))
b = np.zeros(H)

Z = X @ W + b

print("X:", X.shape)  # (32, 10)
print("W:", W.shape)  # (10, 4)
print("b:", b.shape)  # (4,)
print("Z:", Z.shape)  # (32, 4)

5. A Typical Matrix Shape Error

X = np.ones((32, 10))
W = np.ones((8, 4))

# Invalid:
# Z = X @ W

The attempted operation is:

The inner dimensions are 10 and 8. Because they do not match, the multiplication has no valid matrix interpretation under the intended design.

6. Broadcasting Errors Need Right-to-Left Reasoning

For element-wise operations, NumPy compares shapes from the trailing dimensions. Two dimensions are compatible if they are equal or one of them is 1.

A = np.ones((32, 4))
b_good = np.ones((4,))
b_bad = np.ones((32,))

print((A + b_good).shape)  # (32, 4)

# Invalid because trailing dimensions compare as 4 and 32:
# A + b_bad

If the intention is one adjustment per row, reshape explicitly:

row_adjustment = np.ones(32).reshape(32, 1)
result = A + row_adjustment
print(result.shape)  # (32, 4)

7. Rank Errors: (n,), (1,n) and (n,1) Are Different

x = np.arange(5)
row = x.reshape(1, 5)
column = x.reshape(5, 1)

print(x.shape)       # (5,)
print(row.shape)     # (1, 5)
print(column.shape)  # (5, 1)

8. Shape Assertions as Executable Contracts

def linear_forward(X, W, b):
    assert X.ndim == 2, f"X must be rank 2, got {X.shape}"
    assert W.ndim == 2, f"W must be rank 2, got {W.shape}"
    assert b.ndim == 1, f"b must be rank 1, got {b.shape}"

    assert X.shape[1] == W.shape[0], (
        f"Feature mismatch: X has {X.shape[1]} features, "
        f"but W expects {W.shape[0]}"
    )
    assert W.shape[1] == b.shape[0], (
        f"Output mismatch: W produces {W.shape[1]} values, "
        f"but bias has {b.shape[0]}"
    )

    return X @ W + b

For production-facing validation, use explicit exceptions when checks must remain active even if Python assertions are disabled.

9. Shape Reasoning in PyTorch

import torch
from torch import nn

batch = torch.randn(32, 10)
layer = nn.Linear(in_features=10, out_features=4)
output = layer(batch)

print("Input:", batch.shape)    # torch.Size([32, 10])
print("Output:", output.shape)  # torch.Size([32, 4])

For a linear layer receiving an input whose last dimension is \(F\):

The leading dimensions are preserved while the final feature dimension is mapped from \(F\) to \(H\).

10. Non-Finite Values: NaN and Infinity

A numerical pipeline can have perfectly valid shapes and still become unusable because it produces non-finite values.

11. Detect Non-Finite Values Explicitly

import numpy as np

values = np.array([1.0, np.nan, np.inf, -np.inf, 5.0])

print(np.isnan(values))
print(np.isinf(values))
print(np.isfinite(values))

invalid_mask = ~np.isfinite(values)
print("Invalid positions:", np.where(invalid_mask)[0])
print("Invalid count:", invalid_mask.sum())

12. Non-Finite Values Propagate

If one intermediate value becomes NaN, later vectorized operations frequently propagate it.

x = np.array([1.0, np.nan, 3.0])
y = x * 10 + 5
print(y)  # [15. nan 35.]

This is why debugging backward from only the final output can be difficult. Place checks at meaningful stage boundaries.

13. Common Source: Division by Zero

Standardization is defined as:

If the feature is constant, \(\sigma=0\), so the expression divides by zero.

import numpy as np

x = np.array([5.0, 5.0, 5.0])
mean = x.mean()
std = x.std()

print("mean:", mean)
print("std:", std)

if std == 0:
    raise ValueError("Cannot standardize a constant feature")

z = (x - mean) / std

14. Common Source: Logarithm Outside Its Domain

The real-valued natural logarithm requires a positive argument:

import numpy as np

x = np.array([1.0, 0.0, -1.0])

if np.any(x <= 0):
    raise ValueError("log input must be strictly positive")

result = np.log(x)

15. Common Source: Overflow

Exponentials can grow beyond the representable range of a floating-point dtype.

import numpy as np

x = np.array([1.0, 10.0, 1000.0])
result = np.exp(x)

print(result)
print(np.isfinite(result))

When computing expressions such as softmax, use a numerically stable form. Subtracting the maximum does not change the final normalized ratios because the same factor cancels from numerator and denominator:

def stable_softmax(x):
    x = np.asarray(x, dtype=float)
    shifted = x - np.max(x)
    exp_values = np.exp(shifted)
    return exp_values / exp_values.sum()

print(stable_softmax([1000.0, 1001.0, 1002.0]))

16. Build a Finiteness Guard

import numpy as np


def assert_finite(name, values):
    values = np.asarray(values)
    mask = np.isfinite(values)

    if not mask.all():
        bad_indices = np.argwhere(~mask)
        raise ValueError(
            f"{name} contains non-finite values; "
            f"count={(~mask).sum()}, "
            f"first_indices={bad_indices[:5].tolist()}"
        )


X = np.array([[1.0, 2.0], [3.0, 4.0]])
assert_finite("X", X)

17. Stage-by-Stage Numerical Diagnosis

When a multi-stage computation fails, verify every boundary until the first failure is isolated.

def forward_debug(X, W, b):
    assert_finite("input X", X)
    assert_finite("weights W", W)
    assert_finite("bias b", b)

    linear = X @ W + b
    assert_finite("linear output", linear)

    squared = linear ** 2
    assert_finite("squared output", squared)

    loss = squared.mean()
    assert_finite("loss", np.array([loss]))

    return loss

18. Inconsistent Random-State Use

Randomness appears in data splitting, parameter initialization, data shuffling, dropout, augmentation, sampling and simulation. If these random sources are uncontrolled, two apparently identical experiments can differ.

A seed initializes a pseudo-random number generator. Conceptually:

The same seed and the same sequence of generator calls are intended to produce the same pseudo-random sequence for a given implementation and environment.

19. Use an Explicit NumPy Generator

import numpy as np

rng = np.random.default_rng(42)

sample_a = rng.normal(size=5)
sample_b = rng.normal(size=5)

print(sample_a)
print(sample_b)

Creating another generator with the same seed restarts the sequence:

rng1 = np.random.default_rng(42)
rng2 = np.random.default_rng(42)

print(np.array_equal(rng1.normal(size=5), rng2.normal(size=5)))
# True for these matching calls

20. Same Seed Does Not Mean Every Value Should Be Identical

A generator is stateful. Successive calls consume and advance its state.

rng = np.random.default_rng(42)

first = rng.integers(0, 100, size=5)
second = rng.integers(0, 100, size=5)

print(first)
print(second)  # normally different from first

21. A Common Random-State Design Error

def make_noise(size):
    # Weak design: silently restarts the same stream every call.
    rng = np.random.default_rng(42)
    return rng.normal(size=size)

def make_noise(size, rng):
    return rng.normal(size=size)


rng = np.random.default_rng(42)
noise_a = make_noise(5, rng)
noise_b = make_noise(5, rng)

22. Separate Random Streams When Independence Helps

A complex experiment may benefit from independent random streams for data generation, augmentation or another stochastic component. This reduces accidental coupling caused by changing the number of random draws in an unrelated part of the program.

import numpy as np

root = np.random.SeedSequence(42)
data_seed, augmentation_seed = root.spawn(2)

data_rng = np.random.default_rng(data_seed)
augmentation_rng = np.random.default_rng(augmentation_seed)

data_noise = data_rng.normal(size=5)
augmentation_noise = augmentation_rng.normal(size=5)

23. Random State in Data Splitting

For a machine-learning experiment, the data split is part of the experiment configuration. Changing it can change measured performance even when the model code remains unchanged.

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y
)

Record the split seed alongside the dataset version. A reported metric without its split design may be difficult to reproduce or interpret.

24. PyTorch Random-State Control

PyTorch also contains random operations, including parameter initialization and functions such as random tensor generation.

import random
import numpy as np
import torch

SEED = 42

random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)

print(torch.rand(3))

When an experiment uses several libraries, setting only one library's seed does not control the others. Also remember that deterministic reproducibility can depend on device, algorithm and execution environment, not only the seed.

25. Record the Random-State Assumptions

26. Reproducibility Is More Than a Seed

A seed is necessary for many reproducible experiments, but it is not the complete experiment definition.

Changing package versions, preprocessing, hardware behavior, data ordering or algorithm configuration can affect results even when the seed remains unchanged.

27. Log the Evidence Needed for Diagnosis

import logging

logger = logging.getLogger(__name__)


def log_array_state(name, x):
    x = np.asarray(x)

    logger.info(
        "%s shape=%s dtype=%s finite=%s",
        name,
        x.shape,
        x.dtype,
        bool(np.isfinite(x).all())
    )

    if np.issubdtype(x.dtype, np.number) and np.isfinite(x).any():
        finite = x[np.isfinite(x)]
        logger.info(
            "%s finite_min=%g finite_max=%g",
            name,
            finite.min(),
            finite.max()
        )

For debugging, a compact state summary is usually more useful than printing a million-element tensor.

28. TensorBoard as Numerical Evidence

During model training, record metrics that can reveal when numerical behavior changes. Loss, gradient norms and learning rate are common diagnostics.

from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter("runs/debug_run")

# Inside an epoch loop, after values have been calculated:
writer.add_scalar("Loss/train", float(loss), epoch)
writer.add_scalar("Optimization/learning_rate", learning_rate, epoch)
writer.add_scalar("Diagnostics/gradient_norm", gradient_norm, epoch)

writer.close()

If loss changes from finite values to NaN at a particular step, use that step as the boundary for deeper inspection of batches, activations and gradients.

29. An Integrated Diagnostic Function

import numpy as np


def diagnose_linear_inputs(X, W, b):
    X = np.asarray(X)
    W = np.asarray(W)
    b = np.asarray(b)

    report = {
        "X_shape": X.shape,
        "W_shape": W.shape,
        "b_shape": b.shape,
        "X_finite": bool(np.isfinite(X).all()),
        "W_finite": bool(np.isfinite(W).all()),
        "b_finite": bool(np.isfinite(b).all())
    }

    report["matrix_compatible"] = (
        X.ndim == 2
        and W.ndim == 2
        and X.shape[1] == W.shape[0]
    )

    report["bias_compatible"] = (
        W.ndim == 2
        and b.ndim == 1
        and W.shape[1] == b.shape[0]
    )

    return report


X = np.ones((32, 10))
W = np.ones((10, 4))
b = np.zeros(4)

print(diagnose_linear_inputs(X, W, b))

30. A Repeatable Debugging Procedure

31. Common Diagnostic Mistakes

32. Design Review Checklist

33. Practice Problems

34. Worked Shape Problem

Suppose:

For:

Trace the shapes:

If W_2 were accidentally initialized as (30,4), the second layer would fail because H's final dimension is 16, not 30.