Table of Contents

Feature Scaling

Rumman Ansari June 26, 2026 31 views Subject Details

MACHINE LEARNING

Feature Scaling

Bringing all features to a common scale so Machine Learning models can learn fairly and faster.

What is Feature Scaling?

Feature Scaling is a data preprocessing technique used to standardize or normalize the range of independent features (variables) in a dataset so that no single feature dominates others due to its larger magnitude.

In simple words — Feature Scaling makes sure all features speak the same numerical language.

Why is Feature Scaling Needed?

Consider this dataset:

Age	Salary
25	50000
30	60000
40	80000

Problem Salary values (50,000–80,000) are much larger than Age values (25–40). The model may give more importance to Salary just because of its scale.

Solution Apply Feature Scaling so that both features fall within the same range and contribute equally.

Which Algorithms Need Feature Scaling?

Need Scaling

K-Nearest Neighbors (KNN)
K-Means Clustering
Support Vector Machine (SVM)
Linear / Logistic Regression
Principal Component Analysis (PCA)
Neural Networks / Deep Learning
Gradient Descent based algorithms

Don't Need Scaling

Decision Tree
Random Forest
XGBoost / LightGBM
Naive Bayes

Rule of thumb: Distance-based and gradient-based algorithms always need feature scaling.

Types of Feature Scaling

Normalization (Min-Max Scaling)

Compresses all values into a fixed range, usually [0, 1].

FORMULA

$$ x' = \frac{x - \min(x)}{\max(x) - \min(x)} $$

Example:

If Age = 25, min = 20, max = 50 → x' = (25 - 20) / (50 - 20) = 0.166

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()
df[["Age", "Salary"]] = scaler.fit_transform(df[["Age", "Salary"]])

Best For Neural Networks, Image Data (pixels 0–255), KNN, K-Means.

Disadvantage Very sensitive to outliers.

Standardization (Z-Score Scaling)

Transforms data to have mean = 0 and standard deviation = 1.

FORMULA

$$ z = \frac{x - \mu}{\sigma} $$

Where:

μ = mean of feature
σ = standard deviation of feature

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
df[["Age", "Salary"]] = scaler.fit_transform(df[["Age", "Salary"]])

Best For Linear Regression, Logistic Regression, SVM, PCA, Gradient Descent.

Disadvantage Doesn't restrict values to a fixed range.

Robust Scaling

Uses median and interquartile range (IQR) — making it resistant to outliers.

FORMULA

$$ x' = \frac{x - \text{median}(x)}{IQR} $$

from sklearn.preprocessing import RobustScaler

scaler = RobustScaler()
df[["Age", "Salary"]] = scaler.fit_transform(df[["Age", "Salary"]])

Best For Datasets with outliers.

MaxAbs Scaling

Scales each feature by dividing by its maximum absolute value. Result lies in [-1, 1].

FORMULA

$$ x' = \frac{x}{|x|_{max}} $$

from sklearn.preprocessing import MaxAbsScaler

scaler = MaxAbsScaler()
df[["Age", "Salary"]] = scaler.fit_transform(df[["Age", "Salary"]])

Best For Sparse data (e.g., text data with TF-IDF).

Log Transformation

Reduces skewness by applying a logarithm on the feature values.

FORMULA

$$ x' = \log(x + 1) $$

import numpy as np
df["Salary"] = np.log1p(df["Salary"])

Best For Highly skewed data (e.g., Income, Population).

Comparison of Scaling Techniques

Technique	Range	Outlier Sensitive?	Best Use Case
Min-Max Scaling	[0, 1]	Yes	Neural Networks, KNN, K-Means
Standardization	Mean=0, Std=1	Yes (less)	Regression, SVM, PCA
Robust Scaling	Around 0	No	Data with outliers
MaxAbs Scaling	[-1, 1]	Yes	Sparse data (TF-IDF)
Log Transformation	Reduced skew	No	Highly skewed data

Full Python Example

Prerequisites: Python 3.x, pandas, numpy, and scikit-learn.

pip install pandas numpy scikit-learn

import pandas as pd
from sklearn.preprocessing import MinMaxScaler, StandardScaler

# Sample dataset
data = {
    "Age":    [25, 30, 35, 40, 45],
    "Salary": [50000, 60000, 70000, 80000, 90000]
}
df = pd.DataFrame(data)

print("Original Data:\n", df)

# Min-Max Scaling
mm = MinMaxScaler()
df_minmax = pd.DataFrame(mm.fit_transform(df), columns=df.columns)
print("\nMin-Max Scaled:\n", df_minmax)

# Standardization
std = StandardScaler()
df_std = pd.DataFrame(std.fit_transform(df), columns=df.columns)
print("\nStandardized:\n", df_std)

Output Both Age and Salary now share a similar range — the model can treat them equally.

Before vs After Scaling

Before Scaling	After Scaling (Min-Max)
Age = 25, Salary = 50000	Age = 0.00, Salary = 0.00
Age = 35, Salary = 70000	Age = 0.50, Salary = 0.50
Age = 45, Salary = 90000	Age = 1.00, Salary = 1.00

Real-Life Analogy

Feature Scaling = Fair Race

Imagine a race where one person runs on a 100m track and another on a 10,000m track. It's unfair! Feature Scaling brings everyone to the same track length so the comparison becomes fair.

When to Use Which Technique?

Use Min-Max for Neural Networks and image data.
Use Standardization for linear models and PCA.
Use Robust Scaling when data contains outliers.
Use MaxAbs for sparse datasets like text.
Use Log Transform for highly skewed numerical data.

Common Mistakes to Avoid

Mistake 1 Applying scaling on entire dataset before splitting → causes data leakage.

Mistake 2 Using Min-Max Scaling on data containing outliers.

Mistake 3 Forgetting to scale the test data using the same scaler fitted on training data.

Correct Approach Always do: scaler.fit(X_train) → then scaler.transform(X_test).

Best Practices

Quick Tips for Students

Scale after handling missing values.
Always scale numerical features, never target variables in classification.
Use fit only on training, transform on test.
Keep the same scaler object for prediction on new data.
Tree-based models do not require scaling.

Importance of Feature Scaling

Faster Convergence

Speeds up gradient descent
Reduces training time

Better Accuracy

No feature dominates
Improves model fairness

Equal Contribution

All features equally weighted
Better distance calculations

Stable Model

Reduces numerical instability
Avoids overflow issues

Golden Rule

REMEMBER

Different Scales → Biased Model
Same Scale → Fair & Accurate Model

Key Takeaway

Feature Scaling is a vital preprocessing step that ensures all numerical features are on a similar scale, allowing Machine Learning algorithms to perform fairly, efficiently, and accurately. Choosing the right technique — Min-Max, Standardization, or Robust Scaling — depends on your data and algorithm.