Table of Contents

Data Cleaning Techniques

Rumman Ansari June 26, 2026 18 views Subject Details

MACHINE LEARNING

Data Cleaning Techniques

Transforming messy raw data into clean, consistent, and reliable data for Machine Learning.

What is Data Cleaning?

Data Cleaning is the process of detecting and correcting errors, inconsistencies, missing values, and inaccuracies in raw data to make it suitable for analysis and Machine Learning.

In simple words — Data Cleaning means fixing your data so that machines can use it correctly.

Why is Data Cleaning Important?

Real-world data is rarely perfect — it contains:

Missing values
Duplicate records
Noise and errors
Inconsistent formats
Outliers
Typos and incorrect entries

Without Cleaning

Wrong predictions
Lower accuracy
Biased models
Training errors

With Proper Cleaning

Reliable predictions
Improved accuracy
Fair model behavior
Stable performance

Fact: Data scientists spend nearly 80% of their time cleaning and preparing data.

Common Data Issues

Issue	Example
Missing Values	NaN, NULL, blanks
Duplicate Rows	Same record entered twice
Typos	"Mumabi" instead of "Mumbai"
Inconsistent Format	"01/05/2024" vs "2024-05-01"
Wrong Data Types	Age stored as text "25"
Outliers	Salary = 1,00,00,000 in normal data
Whitespace / Case Issues	" Delhi " vs "delhi"

Data Cleaning Techniques

Handling Missing Values

Detect and treat missing or null entries in your dataset.

Common Strategies:

Drop rows/columns with missing values
Fill with mean/median/mode (imputation)
Use forward fill / backward fill
Predict missing values using ML models

df.dropna(inplace=True)                   # remove missing
df["Age"].fillna(df["Age"].mean(), inplace=True)  # fill with mean

Removing Duplicates

Identify and delete duplicate rows that can bias the model.

print(df.duplicated().sum())   # count duplicates
df.drop_duplicates(inplace=True)

Why Important Removes redundancy and prevents biased training.

Fixing Typos & Inconsistent Values

Standardize spellings, case, and values for uniformity.

df["City"] = df["City"].str.strip().str.lower().replace({
    "mumabi": "mumbai",
    "delih": "delhi"
})

Why Important Avoids creating multiple categories for the same value.

Standardizing Data Formats

Ensure dates, numbers, and text follow a consistent format.

df["Date"] = pd.to_datetime(df["Date"], errors="coerce")
df["Phone"] = df["Phone"].astype(str).str.replace("-", "")

Why Important Inconsistent formats cause errors during analysis and modeling.

Correcting Data Types

Convert columns to appropriate data types (int, float, str, datetime).

df["Age"] = df["Age"].astype(int)
df["Salary"] = df["Salary"].astype(float)
df["JoinDate"] = pd.to_datetime(df["JoinDate"])

Why Important Most ML libraries require correct numeric data types.

Handling Outliers

Detect and treat extreme values using IQR or Z-score methods.

Q1 = df["Salary"].quantile(0.25)
Q3 = df["Salary"].quantile(0.75)
IQR = Q3 - Q1

df = df[(df["Salary"] >= Q1 - 1.5*IQR) & (df["Salary"] <= Q3 + 1.5*IQR)]

Why Important Outliers can distort mean, std, and ML model behavior.

Removing Whitespace & Special Characters

Strip unwanted spaces, tabs, and symbols from text columns.

df["Name"] = df["Name"].str.strip()
df["Name"] = df["Name"].str.replace(r"[^a-zA-Z ]", "", regex=True)

Detecting & Fixing Invalid Values

Identify values that don't make sense — e.g., negative age, future dates, impossible salary.

df = df[df["Age"] > 0]
df = df[df["Salary"] >= 0]

Why Important Maintains data validity and logical correctness.

Handling Inconsistent Categorical Data

Merge or rename inconsistent categories.

df["Gender"] = df["Gender"].replace({
    "M": "Male",
    "F": "Female",
    "m": "Male",
    "f": "Female"
})

Removing Irrelevant Features

Drop columns that don't contribute to the model (e.g., IDs, names).

df.drop(["UserID", "Name"], axis=1, inplace=True)

Why Important Reduces noise, improves training speed and accuracy.

Removing Constant or Low-Variance Columns

Drop columns where all values are the same — they add no value.

df = df.loc[:, df.nunique() > 1]

Text Cleaning (for NLP)

Special cleaning for textual data — remove stopwords, punctuation, lowercase conversion, etc.

import re

df["Review"] = df["Review"].str.lower()
df["Review"] = df["Review"].apply(lambda x: re.sub(r"[^a-z\s]", "", str(x)))

Best For Sentiment analysis, chatbots, NLP tasks.

Summary of Cleaning Techniques

Technique	Purpose	Tool/Method
Missing Value Handling	Fill or remove nulls	fillna(), dropna()
Duplicate Removal	Eliminate repeated rows	drop_duplicates()
Typo Fixing	Standardize text	replace(), str.lower()
Format Standardization	Unify dates/numbers	to_datetime(), astype()
Type Conversion	Fix wrong types	astype()
Outlier Handling	Treat extreme values	IQR, Z-score
Whitespace Removal	Clean text	str.strip()
Invalid Value Fix	Logical correctness	Conditional filtering
Category Standardization	Merge similar values	replace()
Drop Irrelevant Columns	Reduce noise	drop()
Drop Low-Variance	Remove constants	nunique()
Text Cleaning	NLP preprocessing	regex, NLTK

Full Python Example

Prerequisites: Python 3.x, pandas, numpy.

pip install pandas numpy

import pandas as pd
import numpy as np

# Sample dataset
data = {
    "Name":   [" John ", "Anna", "Mike", "Anna", None],
    "Age":    [25, -1, None, 30, 28],
    "Gender": ["M", "Female", "m", "F", "Female"],
    "City":   ["Delhi", "mumabi", "Mumbai", "delhi", "Pune"],
    "Salary": [50000, 60000, 1000000, 65000, None]
}

df = pd.DataFrame(data)

# 1. Remove whitespace
df["Name"] = df["Name"].str.strip()

# 2. Drop rows with missing names
df.dropna(subset=["Name"], inplace=True)

# 3. Remove duplicates
df.drop_duplicates(inplace=True)

# 4. Fix invalid ages
df = df[df["Age"].fillna(0) >= 0]

# 5. Fill missing Age & Salary
df["Age"].fillna(df["Age"].mean(), inplace=True)
df["Salary"].fillna(df["Salary"].median(), inplace=True)

# 6. Standardize Gender
df["Gender"] = df["Gender"].replace({"M": "Male", "F": "Female", "m": "Male"})

# 7. Standardize City
df["City"] = df["City"].str.lower().replace({"mumabi": "mumbai"})

# 8. Handle Salary outlier (cap)
df["Salary"] = df["Salary"].clip(upper=200000)

print(df)

Output The dataset is now clean, consistent, and ML-ready.

Before vs After Cleaning

Before

Name	Age	Gender	City	Salary
John	25	M	Delhi	50000
Anna	-1	Female	mumabi	60000
Mike	NaN	m	Mumbai	1000000
Anna	30	F	delhi	65000

After

Name	Age	Gender	City	Salary
John	25	Male	delhi	50000
Anna	30	Female	delhi	65000
Mike	28	Male	mumbai	200000

Real-Life Analogy

Data Cleaning = House Cleaning

Imagine guests are coming home. You wash dishes, throw away spoiled food, fix broken things, and arrange the rooms. Similarly, data cleaning ensures your dataset is "presentable" before the model uses it.

Data Cleaning Workflow

Step-by-Step Process

Inspect the dataset (info, describe, head).
Detect missing values and duplicates.
Fix typos and inconsistent formats.
Correct data types.
Handle outliers and invalid data.
Drop irrelevant columns.
Validate the cleaned data.

Common Mistakes to Avoid

Mistake 1 Cleaning data after splitting incorrectly — causing inconsistencies between train and test.

Mistake 2 Dropping data without checking the percentage of missing values.

Mistake 3 Removing outliers blindly without understanding whether they are real signals.

Mistake 4 Forgetting to document cleaning steps — making the workflow non-reproducible.

Best Practices

Quick Tips

Always inspect raw data before cleaning.
Keep a backup of the original dataset.
Automate cleaning steps via functions or pipelines.
Use visualizations to detect anomalies.
Validate using df.info() and df.describe() after cleaning.
Maintain consistency across all features.

Importance of Data Cleaning

Better Accuracy

Removes misleading inputs
Improves model decisions

Faster Training

Smaller, cleaner data
Better gradient flow

Avoids Bias

Balanced learning
Reliable predictions

Builds Trust

Reliable analytics
Stronger business decisions

Golden Rule

REMEMBER

Garbage In = Garbage Out
Clean Data = Smart Models

Key Takeaway

Data Cleaning is the heart of any Machine Learning workflow. By detecting and fixing missing values, duplicates, errors, inconsistencies, and outliers, you ensure your model learns from high-quality, trustworthy data — leading to better predictions and stronger insights.