Table of Contents

    Introduction to Regression

    Rumman Ansari June 26, 2026 14 views Subject Details
    MACHINE LEARNING

    Introduction to Regression

    Understanding how Machine Learning predicts continuous values like prices, scores, and trends.

    What is Regression?

    Regression is a type of supervised Machine Learning technique used to predict a continuous numerical value based on input features.

    In simple words — Regression helps the model learn the relationship between input variables (X) and a continuous output variable (Y).

    Simple Example

    Imagine you want to predict a person's salary based on years of experience:

    Experience (Years)Salary (₹)
    125,000
    340,000
    560,000
    780,000
    101,00,000

    A Regression model will learn this relationship and predict salary for new experience values, like 6 years.

    Result Predicted salary for 6 years experience ≈ ₹70,000

    Why is Regression Used?

    Regression is used when the output is a continuous value, such as:

    • Predicting house prices
    • Forecasting stock prices
    • Predicting temperature
    • Estimating sales
    • Predicting exam scores
    • Estimating fuel consumption

    Regression vs Classification

    Aspect Regression Classification
    Output Type Continuous (numbers) Discrete (categories)
    Example Predict salary, price Predict Yes/No, Spam/Not Spam
    Algorithm Examples Linear Regression, Random Forest Regressor Logistic Regression, Decision Tree Classifier
    Evaluation Metric MSE, RMSE, MAE, R² Accuracy, Precision, Recall, F1
    7

    Support Vector Regression (SVR)

    Uses SVM technique for predicting continuous values.

    How Regression Works (Workflow)

    Step-by-Step Process

    • Collect and clean the dataset.
    • Identify input (X) and output (Y) variables.
    • Split the data into train/test sets.
    • Choose a regression algorithm.
    • Train the model on training data.
    • Predict values on test data.
    • Evaluate using metrics (MSE, RMSE, R²).

    Python Example — Simple Linear Regression

    Prerequisites: Python 3.x, pandas, scikit-learn, matplotlib. 
    pip install pandas scikit-learn matplotlib
    import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# Sample dataset
data = {
    "Experience": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    "Salary":     [25000, 30000, 40000, 45000, 55000, 65000, 70000, 80000, 90000, 100000]
}
df = pd.DataFrame(data)

# Split data
X = df[["Experience"]]
y = df["Salary"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train model
model = LinearRegression()
model.fit(X_train, y_train)

# Predict
y_pred = model.predict(X_test)
print("Predicted Salaries:", y_pred)

# Visualize
plt.scatter(X, y, color="blue")
plt.plot(X, model.predict(X), color="red")
plt.xlabel("Experience")
plt.ylabel("Salary")
plt.title("Linear Regression Example")
plt.show()
    Output The model learns the salary trend and predicts new values along the regression line.

    Common Regression Evaluation Metrics

    Metric Formula Meaning
    MAE (Mean Absolute Error) $$ \frac{1}{n} \sum |y - \hat{y}| $$ Average absolute error
    MSE (Mean Squared Error) $$ \frac{1}{n} \sum (y - \hat{y})^2 $$ Penalizes larger errors more
    RMSE (Root Mean Squared Error) $$ \sqrt{MSE} $$ Same unit as target value
    R² Score $$ 1 - \frac{SS_{res}}{SS_{tot}} $$ How well model explains variance
    Tip A higher R² (close to 1) and lower RMSE means a better regression model.

    Real-Life Analogy

    Regression = Predicting Tomorrow's Temperature

    Just like a weather model predicts tomorrow's temperature using today's data, regression predicts a continuous value (Y) based on input features (X).

    Real-World Applications

    Real Estate

    • Predict property prices
    • Estimate rent values

    Finance

    • Forecast stock prices
    • Risk and credit scoring

    Business

    • Sales forecasting
    • Customer lifetime value

    Healthcare

    • Predict disease progression
    • Estimate treatment outcomes

    Weather

    • Forecast temperature
    • Predict rainfall amount

    Automobile

    • Predict fuel efficiency
    • Maintenance cost estimation

    Advantages of Regression

    • Easy to understand and implement.
    • Provides clear relationships between variables.
    • Works well for continuous predictions.
    • Many algorithms available for different complexities.

    Disadvantages

    Limitation 1 Sensitive to outliers — they distort predictions.
    Limitation 2 Assumes specific relationships (linear, polynomial) — may not fit all data.
    Limitation 3 Can overfit with too many features.

    Common Mistakes to Avoid

    Mistake 1 Using regression for categorical outputs — use classification instead.
    Mistake 2 Ignoring multicollinearity between input features.
    Mistake 3 Not scaling features before training certain regression models.

    Best Practices

    Quick Tips

    • Always check the distribution of target variable.
    • Handle outliers and missing data first.
    • Normalize/standardize features if needed.
    • Split data into train/test sets.
    • Evaluate using multiple metrics (R², RMSE).
    • Use cross-validation for stable performance.

    Importance of Regression in ML

    Predictive Power

    • Forecasts future trends
    • Strong business impact

    Easy Interpretation

    • Shows relationships clearly
    • Useful for decision-making

    Foundation of ML

    • Base of many advanced models
    • Used in neural networks too

    Practical Use

    • Used in nearly every industry
    • Highly versatile technique

    Golden Rule

    REMEMBER
    Continuous Output + Supervised Learning = Regression

    Key Takeaway

    Regression is a fundamental Machine Learning technique used to predict continuous values. It forms the foundation of many real-world applications — from predicting prices to forecasting trends. Understanding regression is the first step toward mastering predictive modeling.

    Machine Learning Course  |  Chapter: Structured Data Regression  |  Topic 1

    Basic Concept of Regression

    Regression tries to find a mathematical function that best fits the data:

    GENERAL FORMULA
    $$ Y = f(X) + \epsilon $$

    Where:

    • Y = predicted output (target)
    • X = input features
    • f(X) = function (model) that maps X to Y
    • ε = error term

    For example, in Linear Regression:

    LINEAR REGRESSION FORMULA
    $$ Y = b_0 + b_1X_1 + b_2X_2 + \dots + b_nX_n $$

    Key Components of Regression

    1

    Dependent Variable (Y)

    The variable we want to predict. Also called the target or output.

    Example: Salary, Price, Temperature.

    2

    Independent Variables (X)

    The input features used for prediction. Also called predictors.

    Example: Years of experience, area of house, study hours.

    3

    Coefficients (Weights)

    Values learned by the model that show the importance of each feature.

    4

    Error (Residual)

    The difference between the predicted value and the actual value.

    Formula: Error = Actual − Predicted

    Types of Regression

    1

    Linear Regression

    Models the relationship between X and Y using a straight line.

    Example: Predict salary from experience.

    2

    Multiple Linear Regression

    Uses multiple input features to predict a single target.

    Example: Predict house price using area, location, rooms.

    3

    Polynomial Regression

    Fits a non-linear curve to the data using polynomial features.

    Example: Predict growth rate that follows a curve.

    4

    Ridge & Lasso Regression

    Regularized regression that prevents overfitting by penalizing large coefficients.

    5

    Logistic Regression

    Despite its name, it's used for classification (Yes/No), not regression.

    6

    Decision Tree & Random Forest Regression