Table of Contents

Hierarchical Clustering

Rumman Ansari June 26, 2026 16 views Subject Details

MACHINE LEARNING

Hierarchical Clustering

Building a tree of clusters — discover natural hierarchies hidden within your data.

What is Hierarchical Clustering?

Hierarchical Clustering is an unsupervised Machine Learning algorithm that builds a hierarchy of clusters in the form of a tree (dendrogram), where similar data points are grouped together step by step.

In simple words — Hierarchical clustering groups data into nested clusters that form a tree-like structure.

Why Hierarchical Clustering?

No need to predefine the number of clusters (K).
Provides a clear hierarchy/dendrogram.
Great for visualizing data structure.
Captures natural data groupings.
Useful when relationships between clusters matter.

Types of Hierarchical Clustering

Agglomerative (Bottom-Up)

Starts with each data point as its own cluster and merges them step by step.

Most commonly used.

Divisive (Top-Down)

Starts with all data in one cluster and splits them step by step.

Less common, computationally heavier.

How Agglomerative Clustering Works

Step-by-Step Process

Treat each data point as its own cluster.
Find the two closest clusters.
Merge them into a single cluster.
Recompute distances between clusters.
Repeat steps 2–4 until all points form one cluster.
Cut the dendrogram at the desired number of clusters.

Distance Measures Used

Euclidean Distance — most common
Manhattan Distance
Cosine Similarity — for text/document data
Hamming Distance — for categorical data

EUCLIDEAN DISTANCE

$$ d(p, q) = \sqrt{\sum_{i=1}^{n} (q_i - p_i)^2} $$

Linkage Methods

Linkage defines how the distance between two clusters is measured.

Single Linkage

Distance between the two closest points of two clusters.

Result: Long, chain-like clusters.

Complete Linkage

Distance between the two farthest points of two clusters.

Result: Compact, tight clusters.

Average Linkage

Average distance between all pairs from two clusters.

Result: Balanced clusters.

Ward's Linkage

Minimizes the variance within clusters.

Result: Highest quality clusters (most popular).

What is a Dendrogram?

A Dendrogram is a tree-like diagram that shows the order in which clusters are merged or split.

Tip: The optimal number of clusters can be found by cutting the dendrogram at the largest vertical distance that doesn't cross any horizontal line.

Real-Life Example

Suppose we have 6 students with their marks:

Student	Math	Science
A	85	90
B	86	92
C	55	60
D	50	58
E	70	75
F	72	78

Hierarchical Clustering will group them as:

Cluster 1: A, B (high scorers)
Cluster 2: C, D (low scorers)
Cluster 3: E, F (average scorers)

Python Example — Hierarchical Clustering

Prerequisites: Python 3.x, numpy, matplotlib, scipy, scikit-learn.

pip install numpy matplotlib scipy scikit-learn

import numpy as np
import matplotlib.pyplot as plt
from scipy.cluster.hierarchy import dendrogram, linkage
from sklearn.cluster import AgglomerativeClustering

# Sample dataset
X = np.array([
    [85, 90], [86, 92],
    [55, 60], [50, 58],
    [70, 75], [72, 78]
])

# Step 1: Create dendrogram
linked = linkage(X, method="ward")

plt.figure(figsize=(8, 5))
dendrogram(linked, labels=["A", "B", "C", "D", "E", "F"])
plt.title("Dendrogram")
plt.xlabel("Students")
plt.ylabel("Distance")
plt.show()

# Step 2: Apply Agglomerative Clustering
model = AgglomerativeClustering(n_clusters=3, linkage="ward")
labels = model.fit_predict(X)
print("Cluster Labels:", labels)

Output The dendrogram shows how students are grouped step by step, and Agglomerative Clustering assigns each one to a cluster.

Cutting the Dendrogram

To choose the number of clusters:

Draw a horizontal line across the longest vertical gap.
Count the number of vertical lines it crosses.
That number is the optimal cluster count.

Agglomerative vs Divisive

Aspect	Agglomerative	Divisive
Approach	Bottom-Up	Top-Down
Starts With	Each point as cluster	All points in one cluster
Operation	Merging	Splitting
Speed	Faster	Slower
Use Case	Most common	Specialized

K-Means vs Hierarchical Clustering

Aspect	K-Means	Hierarchical
Predefine K?	Yes	No
Speed	Fast	Slow
Scalability	Large datasets	Small/medium datasets
Cluster Shape	Spherical	Any shape
Visualization	Scatter plot	Dendrogram
Outlier Sensitive	Yes	Partial

Real-Life Analogy

Hierarchical Clustering = Family Tree

Just like a family tree shows relationships between ancestors, parents, and children, a dendrogram shows how data points are grouped step by step into bigger families (clusters).

Real-World Applications

Bioinformatics

Gene expression analysis
Species classification

Document Clustering

Group similar articles
Topic hierarchies

Market Segmentation

Multi-level customer groups
Targeted strategies

Healthcare

Group diseases & symptoms
Patient categorization

Image Segmentation

Hierarchical pixel grouping
Object recognition

NLP

Topic modeling
Word grouping

Social Networks

Group users by behavior
Community detection

Biology

Plant & animal classification
Evolutionary trees

Advantages

No need to predefine K.
Provides a beautiful dendrogram visualization.
Works for various distance metrics.
Captures hierarchical relationships.
Easy to interpret.

Disadvantages

Limitation 1 Slow for large datasets — high computational cost.

Limitation 2 Sensitive to noise and outliers.

Limitation 3 Once merged or split, decisions can't be undone.

Limitation 4 Difficult to choose the right linkage and metric.

Evaluation Metrics

Metric	Purpose
Silhouette Score	Measure cluster cohesion & separation
Cophenetic Correlation	How well dendrogram preserves distances
Dunn Index	Cluster compactness vs separation
Davies-Bouldin Index	Lower = better clusters

Common Mistakes to Avoid

Mistake 1 Not scaling features before clustering.

Mistake 2 Applying it on huge datasets — leads to slow performance.

Mistake 3 Choosing wrong linkage method.

Mistake 4 Cutting dendrogram arbitrarily without analyzing structure.

Best Practices

Quick Tips

Always scale data before clustering.
Use Ward's linkage for best balance.
Analyze the dendrogram before choosing K.
Visualize using PCA for high-dimensional data.
Avoid using on huge datasets.
Use multiple evaluation metrics.

Importance of Hierarchical Clustering

Tree-Like Structure

Captures relationships
Visual hierarchy

Insight Discovery

Reveals natural grouping
Easy to interpret

No K Required

Choose K from dendrogram
Flexible decision

Domain Friendly

Used in biology, NLP, healthcare
Useful for hierarchical data

Golden Rule

REMEMBER

Merge Step by Step = Build Hierarchy = Hierarchical Clustering

Key Takeaway

Hierarchical Clustering creates a beautiful tree structure (dendrogram) that captures the natural grouping of data — without the need to predefine the number of clusters. It's especially powerful for revealing relationships, nested patterns, and family-like groupings in your dataset.