Table of Contents
Polynomial Regression
MACHINE LEARNING
Polynomial Regression
When data isn't linear — fit a curve instead of a straight line.
What is Polynomial Regression?
Polynomial Regression is a type of regression that models the relationship between input (X) and output (Y) using a polynomial equation instead of a straight line.
In simple words — Polynomial Regression fits a curve to capture non-linear relationships between variables.
Why Polynomial Regression?
Linear Regression fails when data follows a curved pattern. For such cases, a straight line doesn't fit well — we need a curve.
Linear Regression Fails
- Cannot capture curves
- High error in non-linear data
- Underfits the data
Polynomial Regression Works
- Fits curved patterns
- Captures non-linear relationships
- Higher accuracy on complex data
Mathematical Formula
POLYNOMIAL REGRESSION EQUATION
$$ Y = b_0 + b_1X + b_2X^2 + b_3X^3 + \dots + b_nX^n + \epsilon $$
Where:
- Y = predicted output
- X = input feature
- b₀, b₁, ..., bₙ = coefficients
- n = degree of polynomial
- ε = error term
Note: Polynomial Regression is still considered a linear model because it is linear in its coefficients (b₀, b₁, ...). Only the input X is transformed.
Degrees of Polynomial Regression
1
Degree 1 (Linear)
Same as Linear Regression — straight line.
Equation: Y = b₀ + b₁X
2
Degree 2 (Quadratic)
Fits a U-shaped or inverted U-shaped curve.
Equation: Y = b₀ + b₁X + b₂X²
3
Degree 3 (Cubic)
Fits an S-shaped curve — captures complex trends.
Equation: Y = b₀ + b₁X + b₂X² + b₃X³
4
Higher Degrees (4, 5, ...)
Fits very complex curves but risks overfitting.
Visual Intuition
| Degree | Shape | Use Case |
|---|---|---|
| 1 | Straight Line | Linear data |
| 2 | Parabola (U-shape) | Simple curves |
| 3 | S-shape | Smooth transitions |
| 4+ | Wavy curve | Complex relationships |
Real-Life Example
Predicting salary based on experience — but the growth is non-linear:
| Experience (Years) | Salary (₹) |
|---|---|
| 1 | 30,000 |
| 2 | 35,000 |
| 3 | 50,000 |
| 4 | 80,000 |
| 5 | 1,20,000 |
| 6 | 2,00,000 |
Notice the salary doesn't grow linearly — it accelerates with experience. A polynomial curve fits this much better than a straight line.
How Polynomial Regression Works
Step-by-Step Process
- Take the original feature X.
- Generate new features: X², X³, ..., Xⁿ.
- Apply Linear Regression on the new features.
- The model fits a curve to the data.
- Evaluate using R², MSE, RMSE.
Python Example
Prerequisites: Python 3.x, pandas, numpy, scikit-learn, matplotlib.
pip install pandas numpy scikit-learn matplotlibimport numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
# Sample data
X = np.array([[1], [2], [3], [4], [5], [6]])
y = np.array([30000, 35000, 50000, 80000, 120000, 200000])
# Step 1: Transform features to polynomial (degree 2)
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
# Step 2: Apply Linear Regression
model = LinearRegression()
model.fit(X_poly, y)
# Step 3: Predict
y_pred = model.predict(X_poly)
# Visualize
plt.scatter(X, y, color="blue", label="Actual")
plt.plot(X, y_pred, color="red", label="Polynomial Fit")
plt.xlabel("Experience (Years)")
plt.ylabel("Salary")
plt.title("Polynomial Regression (Degree 2)")
plt.legend()
plt.show()
print("Coefficients:", model.coef_)
print("Intercept:", model.intercept_)
Output
The red curve fits the data better than a straight line — capturing the non-linear pattern in salary growth.
Choosing the Right Degree
| Degree | Behavior | Risk |
|---|---|---|
| 1 | Linear fit | Underfitting |
| 2 - 3 | Smooth curves | Good balance |
| 4 - 5 | More flexibility | Mild overfitting |
| 6+ | Very wavy | Strong overfitting |
Tip: Use cross-validation or R² comparison to choose the best degree.
Linear vs Polynomial Regression
| Aspect | Linear Regression | Polynomial Regression |
|---|---|---|
| Shape | Straight line | Curve |
| Equation | Y = b₀ + b₁X | Y = b₀ + b₁X + b₂X² + ... |
| Best For | Linear data | Non-linear data |
| Overfitting Risk | Low | High (with large degree) |
| Complexity | Simple | More complex |
Real-Life Analogy
Polynomial Regression = Curved Road
If a road is straight, a straight line (Linear Regression) is enough. But if the road has hills, curves, and turns, you need a curve (Polynomial Regression) to follow it correctly.
Underfitting vs Overfitting
Underfitting (Degree Too Low)
- Too simple model
- Misses curve patterns
- High training & test error
Good Fit (Right Degree)
- Balances simplicity & complexity
- Low training & test error
- Captures real patterns
Overfitting (Degree Too High)
- Memorizes data
- Low training error
- High test error
Real-World Applications
Economics
- Growth & inflation modeling
- Non-linear pricing trends
Weather Prediction
- Temperature curves
- Rainfall seasonal trends
Biology
- Population growth
- Disease spread modeling
Engineering
- Motion prediction
- Load curve analysis
Chemistry
- Reaction rate modeling
- Concentration vs time
Education
- Predict learning curves
- Performance vs study hours
Advantages
- Captures non-linear relationships.
- Flexible to adjust degree based on need.
- Easy to implement using sklearn.
- Provides better accuracy than linear models for curved data.
Disadvantages
Limitation 1
Very sensitive to overfitting at high degrees.
Limitation 2
Highly affected by outliers.
Limitation 3
Choosing the right degree can be tricky.
Limitation 4
Becomes complex with high-dimensional data.
Common Mistakes to Avoid
Mistake 1
Using very high degrees (8, 10, etc.) — leads to overfitting.
Mistake 2
Not scaling features when using high-degree polynomials.
Mistake 3
Forgetting to compare with Linear Regression baseline.
Best Practices
Quick Tips
- Visualize the data first — check if it's curved.
- Start with degree 2 or 3 — keep it simple.
- Use cross-validation to choose the best degree.
- Apply feature scaling for higher degrees.
- Compare with regularization models (Ridge, Lasso) if overfitting.
- Always check both training & test accuracy.
Importance of Polynomial Regression
Handles Non-Linearity
- Captures curves & trends
- Useful in real-world data
Better Accuracy
- Improves predictions for curves
- Better than linear when data bends
Foundation for Curve Fitting
- Used in scientific modeling
- Engineering simulations
Easy to Apply
- Built-in support in libraries
- Quick to test multiple degrees
Golden Rule
REMEMBER
Straight Line → Linear Regression
Curved Pattern → Polynomial Regression
Curved Pattern → Polynomial Regression
Key Takeaway
Polynomial Regression extends Linear Regression to capture non-linear relationships by adding polynomial terms (X², X³, ...). It's perfect when data follows a curve, but choose the right degree carefully to avoid overfitting or underfitting.