Table of Contents

Dimensionality Reduction

Rumman Ansari June 26, 2026 14 views Subject Details

MACHINE LEARNING

Dimensionality Reduction

Reducing the number of features while preserving the most valuable information in your data.

What is Dimensionality Reduction?

Dimensionality Reduction is the process of reducing the number of input features (variables) in a dataset while keeping as much important information as possible.

In simple words — Dimensionality Reduction compresses high-dimensional data into fewer dimensions without losing meaningful patterns.

Simple Example

Suppose a dataset has 100 features:

Some features are highly correlated.
Some features are irrelevant.
Some features add noise.

Using Dimensionality Reduction, we can compress these 100 features into 10–20 meaningful features that capture the same patterns.

Why is Dimensionality Reduction Needed?

High-dimensional data causes a major problem called the Curse of Dimensionality:

Problems With Too Many Features

Increased training time
Overfitting risk
Difficult to visualize
Memory & storage burden
Redundant information

Benefits After Reduction

Faster training
Lower complexity
Better visualization
Improved generalization
Less noise & redundancy

Golden Rule: Fewer but better features → smarter and faster ML models.

The Curse of Dimensionality

As the number of features grows, the volume of the feature space increases exponentially, making the data extremely sparse. This causes:

Difficulty in finding patterns
Distance-based algorithms (KNN, K-Means) lose meaning
Models require more data to generalize
Increased risk of overfitting

Types of Dimensionality Reduction

Feature Selection

Selecting a subset of the original features without changing them.

Examples: Variance Threshold, Correlation Filter, Chi-Square Test, Mutual Information.

Feature Extraction

Creating new features by transforming the original ones.

Examples: PCA, LDA, t-SNE, Autoencoders.

Popular Dimensionality Reduction Techniques

Principal Component Analysis (PCA)

A statistical technique that transforms features into a smaller set of uncorrelated components that capture maximum variance.

CONCEPT

$$ Z = X \cdot W $$

Where W = eigenvectors representing principal components.

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
X_reduced = pca.fit_transform(X)

Best For Numerical data, visualization, noise reduction.

Disadvantage Components are not interpretable.

Linear Discriminant Analysis (LDA)

A supervised method that maximizes the separation between multiple classes.

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis

lda = LinearDiscriminantAnalysis(n_components=2)
X_reduced = lda.fit_transform(X, y)

Best For Classification problems with labels available.

t-SNE (t-Distributed Stochastic Neighbor Embedding)

A non-linear technique mainly used for visualizing high-dimensional data in 2D or 3D.

from sklearn.manifold import TSNE

tsne = TSNE(n_components=2, random_state=42)
X_reduced = tsne.fit_transform(X)

Best For Visualizing clusters in data.

Disadvantage Slow on large datasets, not used for ML training.

UMAP (Uniform Manifold Approximation and Projection)

A modern, faster alternative to t-SNE that preserves both local and global structure.

pip install umap-learn

import umap

reducer = umap.UMAP(n_components=2)
X_reduced = reducer.fit_transform(X)

Best For Large datasets and visualization tasks.

Autoencoders (Deep Learning)

Neural networks that compress data into a smaller representation, then reconstruct it.

from tensorflow.keras import Model, Input
from tensorflow.keras.layers import Dense

input_layer = Input(shape=(100,))
encoded = Dense(32, activation="relu")(input_layer)
decoded = Dense(100, activation="sigmoid")(encoded)

autoencoder = Model(input_layer, decoded)
autoencoder.compile(optimizer="adam", loss="mse")

Best For Complex non-linear data, image and audio compression.

Feature Selection Methods

Pick only the most useful features instead of transforming them.

Variance Threshold — drop low-variance features
Correlation Filter — drop highly correlated columns
Chi-Square Test — for categorical features
Recursive Feature Elimination (RFE) — uses model-based ranking

from sklearn.feature_selection import VarianceThreshold

selector = VarianceThreshold(threshold=0.1)
X_reduced = selector.fit_transform(X)

Comparison of Techniques

Technique	Type	Best For	Supervised?
PCA	Linear	General ML, noise removal	No
LDA	Linear	Classification problems	Yes
t-SNE	Non-linear	Visualization	No
UMAP	Non-linear	Visualization & clustering	No
Autoencoder	Deep Learning	Complex data	No
Feature Selection	Selection	Tabular data	Both

Full Python Example (Using PCA)

Prerequisites: Python 3.x, pandas, numpy, scikit-learn, matplotlib.

pip install pandas numpy scikit-learn matplotlib

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

# Load dataset
iris = load_iris()
X = iris.data
y = iris.target

# Standardize before PCA
X_scaled = StandardScaler().fit_transform(X)

# Apply PCA (4 features → 2 features)
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)

# Visualize
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y, cmap="viridis")
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.title("PCA on Iris Dataset")
plt.show()

print("Explained Variance:", pca.explained_variance_ratio_)

Output A 4-dimensional dataset is reduced to 2 dimensions while keeping over 95% of useful information.

Visual Intuition

Before Reduction	After Reduction
100+ features, hard to visualize	2–10 features, easy to plot
Slow training	Faster training
Overfitting risk	Better generalization

Real-Life Analogy

Dimensionality Reduction = Packing a Suitcase

Imagine traveling abroad. You can't carry your entire wardrobe — instead, you pick only the most important clothes. Dimensionality Reduction works similarly: it keeps only the most useful information and discards the rest.

When to Use Dimensionality Reduction?

When dataset has too many features (high-dimensional).
When features are correlated.
To visualize data in 2D or 3D.
To speed up training time.
To remove noise and improve generalization.
Before clustering for better cluster separation.

Advantages of Dimensionality Reduction

Faster Models

Less data → faster training
Less memory usage

Higher Accuracy

Removes noisy features
Reduces overfitting

Easy Visualization

Compress to 2D/3D plots
Detect clusters easily

Cleaner Data

Removes redundancy
Reduces correlated features

Disadvantages

Risk 1 May lead to loss of important information if too many dimensions are reduced.

Risk 2 Transformed features (like PCA components) are hard to interpret.

Risk 3 Some techniques (t-SNE, UMAP) can be computationally expensive.

Common Mistakes to Avoid

Mistake 1 Applying PCA without scaling features first.

Mistake 2 Reducing too many dimensions — losing meaningful information.

Mistake 3 Using t-SNE for ML model training (it's only for visualization).

Mistake 4 Applying dimensionality reduction before splitting data — causes data leakage.

Best Practices

Quick Tips

Always scale features before PCA/LDA.
Use explained variance ratio to decide number of components.
Use LDA when class labels are available.
Use t-SNE/UMAP only for visualization.
Apply dimensionality reduction after train-test split.
Combine with feature selection for the best results.

Mathematical View (PCA)

PCA finds new axes (principal components) that maximize variance:

FORMULA

$$ \text{Var}(Z) = W^T \Sigma W $$

Where:

Σ = covariance matrix
W = eigenvectors
Z = projected reduced data

Golden Rule

REMEMBER

Less Dimensions + Same Information = Better ML Model

Key Takeaway

Dimensionality Reduction helps simplify complex datasets by reducing features while preserving essential information. Whether through PCA, LDA, t-SNE, or Autoencoders, the goal is the same — make data smaller, smarter, and faster for Machine Learning.