What are Overfitting and Underfitting?

Overfitting and Underfitting are two major issues that affect a Machine Learning model's ability to make accurate predictions on unseen data. Both relate to how well a model learns patterns from the training dataset.

What is Underfitting?

Underfitting occurs when a model is too simple to capture the underlying patterns of the data. It performs poorly on both training and testing sets.

• Model is too basic.
• Fails to learn from the data.
• High bias, low variance.
• Poor predictions on training and test data.

What is Overfitting?

Overfitting occurs when a model becomes too complex and learns even the noise in the training data. It performs very well on training data but poorly on test data.

• Model is too complex.
• Captures noise instead of pattern.
• Low bias, high variance.
• Poor generalization.

Visual Intuition

Real-Life Analogy

Overfitting vs Underfitting

Causes of Underfitting

• Using overly simple algorithms.
• Too few input features.
• Not enough training time.
• Excessive regularization.
• Wrong model choice (linear for non-linear data).

Causes of Overfitting

• Using overly complex models.
• Too many features.
• Insufficient training data.
• No regularization.
• Training too many epochs.

How to Solve Underfitting

• Use a more complex model.
• Add more features.
• Train for more time.
• Reduce regularization.
• Try ensemble techniques (Boosting).

How to Solve Overfitting

• Use simpler models.
• Apply regularization (L1, L2).
• Use cross-validation.
• Get more training data.
• Use dropout (in deep learning).
• Reduce features using Feature Selection or PCA.
• Use Bagging or Ensemble models.

Connection With Bias-Variance Trade-Off

Mathematical Insight

Where:

• Bias² — error from wrong assumptions (causes underfitting).
• Variance — error from sensitivity to data (causes overfitting).
• Irreducible Error — random noise, cannot be controlled.

Python Example — Visualizing Overfitting and Underfitting

pip install scikit-learn matplotlib

import numpy as np
import matplotlib.pyplot as plt
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

# Sample data
X = np.linspace(0, 10, 30).reshape(-1, 1)
y = np.sin(X).ravel() + np.random.normal(0, 0.3, X.shape[0])

degrees = [1, 4, 15]
plt.figure(figsize=(12, 4))

for i, d in enumerate(degrees):
    model = make_pipeline(PolynomialFeatures(d), LinearRegression())
    model.fit(X, y)
    y_pred = model.predict(X)

    plt.subplot(1, 3, i + 1)
    plt.scatter(X, y, color="blue", label="Data")
    plt.plot(X, y_pred, color="red", label=f"Degree {d}")
    plt.title(f"Degree {d}")
    plt.legend()

plt.suptitle("Underfitting (1) → Balanced (4) → Overfitting (15)")
plt.show()

How to Detect Overfitting and Underfitting

Real-World Examples

Where is This Concept Used?

Common Mistakes to Avoid

Best Practices

Importance of Understanding This Concept

Golden Rule