What are Bias and Variance?

Bias and Variance are two major sources of error in Machine Learning models. Understanding them is critical for building accurate and generalizable models.

What is Bias?

Bias is the error introduced when a model makes oversimplified assumptions about the data. A high-bias model fails to capture the underlying patterns and leads to underfitting.

• Model is too simple.
• Misses important data patterns.
• Performs poorly on training and testing data.
• Causes underfitting.

What is Variance?

Variance is the error introduced when a model is too sensitive to small changes in the training data. A high-variance model captures noise instead of the actual pattern — leading to overfitting.

• Model is too complex.
• Memorizes the training data.
• Performs well on training, poorly on test data.
• Causes overfitting.

Visual Intuition

Mathematical Formula

Where:

• Bias² — error from wrong assumptions.
• Variance — error from sensitivity to data changes.
• Irreducible error — noise that cannot be reduced.

Real-Life Analogy

Bias vs Variance — Comparison

Underfitting vs Overfitting

Why is it a Trade-Off?

Reducing bias usually increases variance, and reducing variance usually increases bias. This is known as the Bias-Variance Trade-off.

• Simple model → high bias, low variance.
• Complex model → low bias, high variance.
• Goal → find the sweet spot.

Causes of High Bias

• Using overly simple algorithms.
• Insufficient training data features.
• Not enough training time.
• Linear models for non-linear problems.

Causes of High Variance

• Using overly complex models.
• Too many features.
• Small datasets.
• Noise in training data.

How to Reduce Bias

• Use more complex models.
• Add new features.
• Increase training time.
• Reduce regularization.
• Use ensemble methods like Boosting.

How to Reduce Variance

• Use simpler models.
• Apply regularization (L1, L2).
• Use more training data.
• Reduce features (Dimensionality Reduction).
• Use ensemble methods like Bagging.

Python Example — Visualizing Bias & Variance

pip install scikit-learn matplotlib

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

# Create sample data
X = np.linspace(0, 10, 30).reshape(-1, 1)
y = np.sin(X).ravel() + np.random.normal(0, 0.3, X.shape[0])

degrees = [1, 4, 15]
plt.figure(figsize=(12, 4))

for i, d in enumerate(degrees):
    model = make_pipeline(PolynomialFeatures(d), LinearRegression())
    model.fit(X, y)
    y_pred = model.predict(X)

    plt.subplot(1, 3, i + 1)
    plt.scatter(X, y, color="blue", label="Data")
    plt.plot(X, y_pred, color="red", label=f"Degree {d}")
    plt.title(f"Degree {d}")
    plt.legend()

plt.suptitle("Bias-Variance Trade-Off")
plt.show()

Bias and Variance in Common ML Models

The Bias-Variance Curve

The total error decreases initially and then increases when the model becomes too complex:

• Bias decreases as model complexity increases.
• Variance increases as complexity increases.
• Optimum point = Minimum total error.

Where is Bias-Variance Used?

Common Mistakes to Avoid

Best Practices

Importance of Bias-Variance Concept

Golden Rule