Linear Regression
Linear Regression
The simplest yet most powerful algorithm to understand relationships between data and make predictions.
Introduction to Linear Regression
Linear Regression is one of the most fundamental algorithms in Machine Learning. It is used to predict a continuous numerical value based on one or more input features. In simple words, it tries to draw the best straight line through the data points that minimizes the difference between predicted and actual values.
Definition
It assumes a linear relationship between input (X) and output (Y).
The Linear Regression Equation
The mathematical formula behind Linear Regression is:
$$ y = mx + c $$
Or in general form:
$$ y = b_0 + b_1x_1 + b_2x_2 + ... + b_nx_n $$
- y → Predicted value (Dependent variable)
- x → Input feature (Independent variable)
- b₀ (c) → Intercept (where the line crosses the Y-axis)
- b₁ (m) → Slope (rate of change of y w.r.t. x)
How Linear Regression Works
- Collect data points (X, Y).
- Plot them on a graph.
- Draw the best line through them.
- Calculate prediction errors (residuals).
- Adjust the slope and intercept to minimize the total error.
Types of Linear Regression
Simple Linear Regression
- One input feature (X).
- One output (Y).
- Example: Predicting salary based on experience.
Multiple Linear Regression
- Multiple input features (X1, X2, X3...).
- One output (Y).
- Example: Predicting house price using area, rooms, location.
Polynomial Regression
- Used when relationship is non-linear.
- Equation: y = b₀ + b₁x + b₂x² + b₃x³...
Ridge & Lasso Regression
- Used to handle overfitting and multicollinearity.
- Apply regularization techniques.
Cost Function & Error Minimization
To find the best-fit line, Linear Regression minimizes the Mean Squared Error (MSE):
$$ MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 $$
- yᵢ → Actual value
- ŷᵢ → Predicted value
- n → Number of samples
Gradient Descent (Optimization)
Gradient Descent updates the slope (m) and intercept (c) step by step to minimize the cost function.
$$ m = m - \alpha \frac{\partial J}{\partial m}, \quad c = c - \alpha \frac{\partial J}{\partial c} $$
- α → Learning rate (step size)
- J → Cost function (error)
Assumptions of Linear Regression
Linearity
The relationship between X and Y must be linear.
Independence
Observations should be independent of each other.
Homoscedasticity
Residuals should have constant variance.
Normality
Errors should follow a normal distribution.
Simple Example in Python
Predicting marks based on study hours using Linear Regression:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
# Sample dataset
data = {
"Hours": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
"Marks": [15, 25, 35, 45, 55, 65, 75, 85, 95, 100]
}
df = pd.DataFrame(data)
# Features and target
X = df[["Hours"]]
y = df["Marks"]
# Train-Test Split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train the model
model = LinearRegression()
model.fit(X_train, y_train)
# Predict
y_pred = model.predict(X_test)
print("Predicted Marks:", y_pred)
# Coefficients
print("Slope (m):", model.coef_[0])
print("Intercept (c):", model.intercept_)
# Plot
plt.scatter(X, y, color='blue')
plt.plot(X, model.predict(X), color='red')
plt.xlabel("Study Hours")
plt.ylabel("Marks")
plt.title("Linear Regression Example")
plt.show()
Performance Metrics for Linear Regression
| Metric | Description | Formula |
|---|---|---|
| MAE | Mean Absolute Error | $$ \frac{1}{n}\sum|y - \hat{y}| $$ |
| MSE | Mean Squared Error | $$ \frac{1}{n}\sum(y - \hat{y})^2 $$ |
| RMSE | Root Mean Squared Error | $$ \sqrt{MSE} $$ |
| R² Score | Coefficient of Determination | $$ 1 - \frac{SS_{res}}{SS_{tot}} $$ |
Simple vs Multiple Linear Regression
| Aspect | Simple | Multiple |
|---|---|---|
| Inputs | 1 feature | Multiple features |
| Equation | y = mx + c | y = b₀ + b₁x₁ + b₂x₂ + ... |
| Use Case | Simple prediction | Complex prediction |
| Example | Marks vs Hours | Price = f(Area, Rooms, Location) |
Advantages vs Limitations
Advantages
- Simple and easy to implement.
- Provides clear interpretability.
- Works well with small datasets.
- Foundation for advanced models.
Limitations
- Assumes linear relationship only.
- Sensitive to outliers.
- Poor with complex datasets.
- Cannot capture non-linearity easily.
Real-World Applications
Finance
- Predict stock prices.
- Estimate insurance risk.
Real Estate
- Predict house prices based on features.
Healthcare
- Predict patient recovery time.
- Estimate disease progression.
E-Commerce
- Forecast sales and demand.
- Predict customer spend.
Common Mistakes vs Best Practices
| Common Mistakes | Best Practices |
|---|---|
| Using non-linear data directly | Check linearity using scatter plots |
| Ignoring outliers | Detect and handle outliers properly |
| Skipping feature scaling | Normalize features before training |
| Overfitting on small datasets | Use cross-validation |
Prerequisites Before Learning
What You Should Know First
- Basic algebra and equations.
- Understanding of statistics (mean, variance).
- Knowledge of Python and NumPy.
- Familiarity with Pandas and scikit-learn.
- Basic ML workflow understanding.
Pro Tips to Master Linear Regression
Smart Strategy
- Visualize data before applying regression.
- Use scatter plots to check linearity.
- Always evaluate using R² and RMSE.
- Apply regularization (Ridge/Lasso) for overfitting.
- Combine domain knowledge with statistics for accuracy.
Key Formulas Cheat Sheet
Final Takeaway
Linear Regression is the foundation of every prediction model. It's simple, interpretable, and incredibly useful. Master it once — and you'll easily understand advanced ML algorithms built on top of it 🚀.