Table of Contents

Handling Outliers

Rumman Ansari June 26, 2026 16 views Subject Details

MACHINE LEARNING

Handling Outliers

Detecting, analyzing, and managing extreme values that can mislead your Machine Learning models.

What is an Outlier?

An Outlier is a data point that is significantly different from the rest of the dataset. It lies far away from the majority of observations and can distort statistical analysis and ML model performance.

In simple words — An outlier is a value that "doesn't fit" with the rest of your data.

Simple Example

Consider this dataset of student marks:

Student	Marks
A	78
B	82
C	85
D	88
E	5

Outlier Detected Student E with marks 5 is clearly an outlier compared to others (78–88).

Causes of Outliers

Human Errors — typos during data entry.
Measurement Errors — faulty sensors or devices.
Sampling Errors — collecting data from wrong groups.
Natural Variations — genuine extreme values (e.g., billionaires' income).
Data Integration — combining data from multiple sources.

Types of Outliers

Univariate Outliers

Outliers found in one variable only.

Example: A salary value of ₹10,00,00,000 in a regular employee dataset.

Multivariate Outliers

Outliers detected by combining multiple variables.

Example: A person aged 25 having 40 years of experience.

Global Outliers

Values that are extremely different from the entire dataset.

Example: Temperature of 80°C recorded in a city's daily weather data.

Contextual Outliers

Values that are unusual in a specific context.

Example: 30°C in winter — normal in summer but unusual in December.

Why Handle Outliers?

If Not Handled

Distort mean and standard deviation
Mislead the ML model
Skewed predictions
Wrong feature importance

If Properly Handled

Accurate model training
Reliable predictions
Better statistical analysis
Improved data quality

Important: Not all outliers are "bad". Some represent genuine extreme values that may be meaningful (e.g., fraud detection).

Methods to Detect Outliers

Boxplot Method (Visual)

Uses a box-and-whisker plot to detect values outside the whiskers.

import seaborn as sns
import matplotlib.pyplot as plt

sns.boxplot(x=df["Salary"])
plt.show()

Best For Quick visual inspection of outliers.

IQR (Interquartile Range) Method

Statistical technique using quartiles to detect outliers.

FORMULA

$$ IQR = Q3 - Q1 $$ $$ \text{Lower Bound} = Q1 - 1.5 \times IQR $$ $$ \text{Upper Bound} = Q3 + 1.5 \times IQR $$

Any value below the Lower Bound or above the Upper Bound is considered an outlier.

Q1 = df["Salary"].quantile(0.25)
Q3 = df["Salary"].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

outliers = df[(df["Salary"] < lower) | (df["Salary"] > upper)]
print(outliers)

Best For Non-normal data with skewness.

Z-Score Method

Detects how many standard deviations a value is from the mean.

FORMULA

$$ z = \frac{x - \mu}{\sigma} $$

Generally, values with |z| > 3 are considered outliers.

from scipy import stats
import numpy as np

z = np.abs(stats.zscore(df["Salary"]))
outliers = df[z > 3]
print(outliers)

Best For Normally distributed numerical data.

Scatter Plot

Visual method to detect outliers in two variables.

plt.scatter(df["Age"], df["Salary"])
plt.xlabel("Age")
plt.ylabel("Salary")
plt.show()

Best For Visualizing bivariate relationships.

Machine Learning Methods

Advanced techniques using algorithms.

Isolation Forest — isolates anomalies using trees.
DBSCAN — density-based clustering treats outliers as noise.
Local Outlier Factor (LOF) — measures local density deviation.

from sklearn.ensemble import IsolationForest

iso = IsolationForest(contamination=0.05)
df["outlier"] = iso.fit_predict(df[["Salary"]])
# -1 means outlier, 1 means normal

Best For Complex multivariate data and large datasets.

Techniques to Handle Outliers

Removing Outliers

Drop the rows containing outlier values from the dataset.

df_clean = df[(df["Salary"] >= lower) & (df["Salary"] <= upper)]

When to Use When outliers are due to errors and the dataset is large.

Avoid When Outliers carry meaningful information (e.g., fraud, anomalies).

Capping (Winsorization)

Replace extreme values with upper/lower thresholds instead of deleting them.

df["Salary"] = df["Salary"].clip(lower, upper)

Why Useful Preserves data size while reducing the impact of extreme values.

Transformation

Apply mathematical transformations to reduce skewness caused by outliers.

Log Transformation — best for right-skewed data.
Square Root Transformation — softer effect than log.
Box-Cox / Yeo-Johnson — power-based transformations.

import numpy as np

df["Salary_log"] = np.log1p(df["Salary"])

Imputation

Replace outliers with statistical values like mean, median, or mode.

median_value = df["Salary"].median()
df.loc[df["Salary"] > upper, "Salary"] = median_value

Best For Small datasets where data loss must be avoided.

Use Robust Algorithms

Some algorithms are less affected by outliers — use them when removal isn't ideal.

Decision Trees
Random Forest
XGBoost / LightGBM
Robust Scaler in preprocessing

Comparison of Techniques

Technique	Advantage	Disadvantage
Removal	Simple & clean	Loses data
Capping	Preserves data size	May reduce variance
Transformation	Reduces skewness	Changes interpretation
Imputation	No data loss	Adds bias
Robust Algorithms	No data changes	Algorithm-specific

Full Python Example

Prerequisites: Python 3.x, pandas, numpy, matplotlib, scikit-learn.

pip install pandas numpy matplotlib scikit-learn

import pandas as pd
import numpy as np

# Sample dataset
data = {"Salary": [25000, 30000, 32000, 35000, 36000, 40000, 42000, 1000000]}
df = pd.DataFrame(data)

# Step 1: Detect using IQR
Q1 = df["Salary"].quantile(0.25)
Q3 = df["Salary"].quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

print("Lower:", lower, "Upper:", upper)

# Step 2: Detect outliers
outliers = df[(df["Salary"] < lower) | (df["Salary"] > upper)]
print("Outliers:\n", outliers)

# Step 3: Handle outliers using Capping
df["Salary"] = df["Salary"].clip(lower, upper)

print("Cleaned Data:\n", df)

Output The huge outlier value (1,000,000) is now capped, making the dataset balanced and ML-ready.

Before vs After Handling

Before	After
Mean = 1,40,000 (distorted)	Mean = 34,000 (realistic)
Salary = 10,00,000 (outlier)	Salary = capped at upper bound
Misleading patterns	Clean & balanced patterns

Real-Life Analogy

Outliers = The Odd One Out

Imagine measuring the height of 10 students aged 12. If one is 6'5", they are the outlier. They are not "wrong" — but they don't represent the typical student. Handling outliers helps the model understand what's normal.

When Should You KEEP Outliers?

When detecting fraud or anomalies.
In medical/scientific data where rare values matter.
When dataset is small and removing reduces useful information.
When outliers reflect genuine real-world events.

Common Mistakes to Avoid

Mistake 1 Removing every extreme value without analyzing whether it's a real or error value.

Mistake 2 Applying Z-score on highly skewed data — it works only for normal distributions.

Mistake 3 Handling outliers before splitting data — causes data leakage.

Correct Approach Always handle outliers after train-test split, using only the training data parameters.

Best Practices

Quick Tips

Always visualize data first (boxplot, scatter plot).
Understand the cause before removing outliers.
Use IQR for skewed data and Z-Score for normal data.
Prefer capping or transformation over deletion.
Use Robust Scaler if you must keep outliers.
Document all outlier-handling decisions for reproducibility.

Importance of Handling Outliers

Improves Accuracy

Removes misleading patterns
Better predictions

Stable Statistics

Accurate mean & std
Reliable distributions

Faster Training

Cleaner gradient updates
Better convergence

Avoids Bias

Prevents skewed learning
Fair model behavior

Golden Rule

REMEMBER

Detect → Analyze → Decide → Handle

Key Takeaway

Outliers can either be errors or important signals. The goal of handling outliers is not to blindly remove them, but to detect, understand, and treat them properly to build a Machine Learning model that performs reliably on real-world data.