Table of Contents

Handling Missing Data

Rumman Ansari June 26, 2026 19 views Subject Details

MACHINE LEARNING

Handling Missing Data

Learn how to detect, analyze, and fix missing values to make your dataset ML-ready.

What is Missing Data?

Missing Data refers to the values that are not recorded, not available, or lost in a dataset. These appear as NaN, NULL, None, blank cells, or placeholder values like -1, ?, or "unknown".

Missing data is one of the most common problems in real-world datasets, and how we handle it directly impacts the accuracy and reliability of a Machine Learning model.

Why Does Missing Data Occur?

Human Error — survey skipped or wrong entry.
System Failure — sensor or device malfunction.
Privacy Reasons — users refuse to share info.
Data Merging Issues — combining multiple datasets.
Corrupted Files — data lost during transfer.

Types of Missing Data

MCAR — Missing Completely At Random

Missing values have no relationship with any variable.

Example: A few rows lost due to a system crash.

MAR — Missing At Random

Missing values depend on other observed variables but not on the missing value itself.

Example: Women skipping weight in a survey more than men.

MNAR — Missing Not At Random

Missing values depend on the missing value itself.

Example: People with very high salary not disclosing income.

Why Handle Missing Data?

If Not Handled

Model gives errors (NaN propagation)
Reduces accuracy
Causes biased predictions
Some algorithms refuse to train

If Properly Handled

Stable training process
Better predictions
Realistic data distribution
Trustworthy results

How to Detect Missing Data?

In Python, we usually use pandas to check for missing values.

Prerequisites: Python 3.x and the pandas library.

pip install pandas numpy scikit-learn

import pandas as pd

df = pd.read_csv("data.csv")

# Check missing values
print(df.isnull().sum())     # Count of missing per column
print(df.isnull().mean()*100) # Percentage of missing per column

Tip Always check missing data before training a model. A column with more than 50% missing values is usually a candidate to drop.

Techniques to Handle Missing Data

Deletion Method

Remove rows or columns containing missing values.

a) Row Deletion (Listwise Deletion)

Remove the entire row that contains a missing value.

df.dropna(inplace=True)

When to Use When missing rows are very few (e.g., less than 5%).

Disadvantage Causes data loss and may introduce bias.

b) Column Deletion

Drop a whole column if too many values are missing.

df.drop("ColumnName", axis=1, inplace=True)

When to Use When more than 50–60% of the column is missing.

Imputation Method

Replace missing values with a calculated/estimated value.

a) Mean / Median / Mode Imputation

Used for numerical and categorical data.

# Mean
df["Age"] = df["Age"].fillna(df["Age"].mean())

# Median
df["Salary"] = df["Salary"].fillna(df["Salary"].median())

# Mode (for categorical)
df["City"] = df["City"].fillna(df["City"].mode()[0])

Method	Best For	Avoid When
Mean	Numerical, normally distributed data	Data has outliers
Median	Numerical with outliers/skewed data	Balanced data
Mode	Categorical data	High variety categories

b) Constant Value Imputation

Replace missing values with a fixed value such as 0 or "Unknown".

df["Gender"] = df["Gender"].fillna("Unknown")

c) Forward Fill / Backward Fill

Used mainly for time series data.

df.fillna(method="ffill", inplace=True)   # forward fill
df.fillna(method="bfill", inplace=True)   # backward fill

d) KNN Imputation (Advanced)

Estimates missing values using the K-Nearest Neighbors algorithm.

from sklearn.impute import KNNImputer
imputer = KNNImputer(n_neighbors=3)
df_filled = imputer.fit_transform(df)

Pro Tip KNN gives more accurate imputation than mean/median in many real-world cases.

Prediction-Based Imputation

Train a small ML model to predict the missing values using other columns.

Example: Use Linear Regression to predict missing Salary based on Age and Experience.

Advantage Produces the most realistic, data-driven results.

Disadvantage Computationally expensive and complex.

Indicator / Flag Variable

Add a new column indicating which values were originally missing.

df["Age_missing"] = df["Age"].isnull().astype(int)
df["Age"] = df["Age"].fillna(df["Age"].mean())

Why Useful Helps the model learn the fact that the value was missing.

Comparison of Techniques

Technique	Type	Advantage	Disadvantage
Row/Column Deletion	Removal	Simple & fast	Loss of data
Mean / Median / Mode	Statistical	Easy to implement	Ignores relationships
Forward / Backward Fill	Sequential	Great for time-series	Not good for random data
KNN Imputation	ML-based	Accurate predictions	Slow for big data
Predictive Modeling	ML-based	Most realistic	Complex setup
Flag Indicator	Hybrid	Preserves missingness info	Adds extra column

Full Example — Handling Missing Data in Python

import pandas as pd
from sklearn.impute import SimpleImputer

# Sample data
data = {
    "Name":   ["John", "Anna", "Mike", "Sara", "Tom"],
    "Age":    [25, None, 30, None, 22],
    "Salary": [50000, 60000, None, 65000, 40000],
    "City":   ["Delhi", "Mumbai", None, "Pune", "Delhi"]
}
df = pd.DataFrame(data)

# Step 1: Detect missing values
print(df.isnull().sum())

# Step 2: Fill numerical columns with mean
num_imputer = SimpleImputer(strategy="mean")
df[["Age", "Salary"]] = num_imputer.fit_transform(df[["Age", "Salary"]])

# Step 3: Fill categorical column with mode
df["City"] = df["City"].fillna(df["City"].mode()[0])

print(df)

Output A clean dataset with no missing values, ready for ML training.

Mathematical View

The most common imputation formulas:

MEAN IMPUTATION

$$ \bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i $$

MEDIAN IMPUTATION

$$ \text{Median} = \text{Middle value of sorted } x $$

MODE IMPUTATION

$$ \text{Mode} = \text{Most frequent value in } x $$

Real-Life Analogy

Missing Data = Missing Puzzle Pieces

If a few puzzle pieces are missing, you can guess (imputation) based on the surrounding pieces. If too many are missing, the puzzle becomes useless and must be discarded (deletion).

Best Practices

Quick Rules to Remember

Always analyze the missing data pattern before fixing.
Use median for skewed numerical data, mean for normal data.
Use mode for categorical columns.
Drop columns only if more than 50% missing.
Always perform imputation after splitting train/test to avoid data leakage.
For time-series, use ffill or bfill.

Common Mistakes to Avoid

Mistake 1 Filling missing values before splitting data into train/test — causes data leakage.

Mistake 2 Using mean on data with many outliers — leads to incorrect imputation.

Mistake 3 Dropping rows blindly without checking what percentage is missing.

Key Takeaway

Handling Missing Data is a critical preprocessing step that ensures your dataset is complete, consistent, and reliable. Always choose the right technique based on the type and amount of missingness — and never ignore it before training a model.