What is Lasso Regression?

Lasso Regression (Least Absolute Shrinkage and Selection Operator) is a type of regularized linear regression that adds an L1 penalty to the cost function. It shrinks coefficients and can make some of them exactly zero, effectively performing feature selection.

Why Lasso Regression?

Standard Linear Regression can struggle with:

Mathematical Formula

Standard Linear Regression Loss:

Lasso Regression Loss (with L1 penalty):

Where:

• yᵢ = actual value
• ŷᵢ = predicted value
• |bⱼ| = absolute value of feature coefficient
• λ (lambda / alpha) = regularization strength

How Lasso Performs Feature Selection

The L1 penalty creates sharp corners in the cost function. When the optimizer reaches a corner, some coefficients become exactly zero, effectively removing those features from the model.

• Small α → fewer features eliminated.
• Large α → many features eliminated.
• Very large α → all coefficients → 0 (model fails).

Role of α (Alpha / Lambda)

How Lasso Regression Works

Python Example

pip install pandas numpy scikit-learn

import pandas as pd
from sklearn.linear_model import Lasso
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

# Sample dataset
data = {
    "Area":     [1000, 1500, 2000, 2500, 3000, 3500, 4000],
    "Bedrooms": [2, 3, 3, 4, 4, 5, 5],
    "Age":      [10, 5, 15, 8, 3, 2, 1],
    "Garden":   [0, 1, 0, 1, 1, 0, 1],
    "Price":    [40, 55, 65, 80, 100, 120, 140]
}
df = pd.DataFrame(data)

X = df[["Area", "Bedrooms", "Age", "Garden"]]
y = df["Price"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Lasso Regression
lasso = Lasso(alpha=1.0)
lasso.fit(X_train, y_train)

# Predict
y_pred = lasso.predict(X_test)

print("Coefficients:", lasso.coef_)
print("Intercept:", lasso.intercept_)
print("MSE:", mean_squared_error(y_test, y_pred))
print("R²:", r2_score(y_test, y_pred))

Comparing Different Alpha Values

for alpha in [0.01, 0.1, 1, 10, 100]:
    model = Lasso(alpha=alpha)
    model.fit(X_train, y_train)
    print(f"Alpha={alpha}, Coefficients={model.coef_}")

Linear vs Lasso Regression

Ridge vs Lasso Regression

Bias-Variance Tradeoff

Real-Life Analogy

Real-World Applications

Advantages

• Performs automatic feature selection.
• Reduces overfitting.
• Simplifies models with many features.
• Easier to interpret due to fewer features.
• Great for high-dimensional datasets.

Disadvantages

Common Mistakes to Avoid

Auto-Tuning with LassoCV

from sklearn.linear_model import LassoCV

lasso_cv = LassoCV(alphas=[0.01, 0.1, 1, 10, 100], cv=5)
lasso_cv.fit(X_train, y_train)

print("Best Alpha:", lasso_cv.alpha_)
print("Best Score:", lasso_cv.score(X_test, y_test))

Best Practices

Importance of Lasso Regression

Golden Rule