Table of Contents

Encoding Categorical Data

Rumman Ansari June 26, 2026 34 views Subject Details

MACHINE LEARNING

Encoding Categorical Data

Converting text labels into numerical values that Machine Learning models can understand.

What is Encoding Categorical Data?

Encoding Categorical Data is the process of converting non-numeric (text) data into numerical form so that Machine Learning models can understand and process it.

Machine Learning models work only with numbers — so categorical values like "Red", "Blue", "Male", "Female" must be converted into numerical form before training.

What is Categorical Data?

Categorical Data represents discrete values that fall into specific groups or categories.

Nominal Data

No inherent order or ranking between categories.

Examples: Colors (Red, Blue, Green), Gender (Male, Female), Cities (Delhi, Mumbai, Pune).

Ordinal Data

Categories have a meaningful order or ranking.

Examples: Education (High School < Bachelor < Master), Size (Small < Medium < Large), Rating (Poor < Good < Excellent).

Why Encoding is Needed?

Without Encoding

ML models throw errors
Cannot perform mathematical operations
Algorithm fails to train
No way to compute distances

With Encoding

Data becomes numeric
Algorithms train smoothly
Distances and gradients can be computed
Improves model performance

Golden Rule: All categorical data must be encoded into numbers before model training.

Techniques of Encoding Categorical Data

Label Encoding

Assigns a unique integer to each category. Best suited for ordinal data.

Example:

Category	Encoded Value
Small	0
Medium	1
Large	2

from sklearn.preprocessing import LabelEncoder

le = LabelEncoder()
df["Size_encoded"] = le.fit_transform(df["Size"])

Best For Ordinal data (data with order).

Avoid When Used on nominal data — may make the model think 2 > 1 > 0, which is misleading.

One-Hot Encoding (OHE)

Creates a separate binary column (0/1) for each category. Best for nominal data.

Example:

Color	Red	Blue	Green
Red	1	0	0
Blue	0	1	0
Green	0	0	1

import pandas as pd

df = pd.get_dummies(df, columns=["Color"])

Or using scikit-learn:

from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(sparse_output=False)
encoded = ohe.fit_transform(df[["Color"]])

Best For Nominal data (no inherent order).

Disadvantage Creates many columns when categories are large — curse of dimensionality.

Ordinal Encoding

Used when categorical data has a natural order that we manually define.

from sklearn.preprocessing import OrdinalEncoder

oe = OrdinalEncoder(categories=[["Low", "Medium", "High"]])
df["Priority_encoded"] = oe.fit_transform(df[["Priority"]])

Best For Data like Education Level, Ratings, Priority, etc.

Binary Encoding

Converts categories into binary numbers and splits them into separate columns. Reduces dimensionality compared to OHE.

Example: "Red" → 1 → 01, "Blue" → 2 → 10, "Green" → 3 → 11

pip install category_encoders

import category_encoders as ce

encoder = ce.BinaryEncoder(cols=["City"])
df_encoded = encoder.fit_transform(df)

Best For High-cardinality categorical features (many unique values).

Frequency / Count Encoding

Replaces each category with the frequency (or count) of its occurrence in the dataset.

df["City_encoded"] = df["City"].map(df["City"].value_counts())

Best For High-cardinality categorical features.

Disadvantage Two different categories with same frequency get same encoding.

Target / Mean Encoding

Replaces each category with the mean of the target variable for that category.

mean_encoded = df.groupby("City")["Target"].mean()
df["City_encoded"] = df["City"].map(mean_encoded)

Best For Regression/classification with high-cardinality features.

Disadvantage Risk of data leakage — must compute only on training data.

Hashing Encoding

Applies a hash function to map categories into a fixed number of columns.

from sklearn.feature_extraction import FeatureHasher

hasher = FeatureHasher(n_features=8, input_type="string")
hashed = hasher.transform(df["City"])

Best For Very large datasets with millions of unique values.

Comparison of Encoding Techniques

Technique	Best For	Output	Pros	Cons
Label Encoding	Ordinal Data	Integer	Simple, fast	Adds false ranking to nominal data
One-Hot Encoding	Nominal Data	Binary columns	No false order	Increases columns
Ordinal Encoding	Ordered categories	Integer	Preserves order	Manual mapping needed
Binary Encoding	High-cardinality	Binary columns	Compact	Less interpretable
Frequency Encoding	High-cardinality	Integer/Float	Easy	Same freq → same encoding
Target Encoding	Supervised tasks	Float	Powerful	Data leakage risk
Hashing Encoding	Massive datasets	Hashed columns	Scalable	Collisions possible

Full Python Example

Prerequisites: Python 3.x, pandas, and scikit-learn.

pip install pandas scikit-learn

import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder

# Sample dataset
data = {
    "Name":   ["John", "Anna", "Mike", "Sara"],
    "Gender": ["Male", "Female", "Male", "Female"],
    "Size":   ["Small", "Medium", "Large", "Medium"]
}
df = pd.DataFrame(data)

# Label Encoding for ordinal column
le = LabelEncoder()
df["Size_encoded"] = le.fit_transform(df["Size"])

# One-Hot Encoding for nominal column
df = pd.get_dummies(df, columns=["Gender"])

print(df)

Output Categorical columns are now fully numeric and ready for ML training.

Before vs After Encoding

Before (Raw Data)

Name	Gender	Size
John	Male	Small
Anna	Female	Medium
Mike	Male	Large

After Encoding

Name	Size_encoded	Gender_Male	Gender_Female
John	0	1	0
Anna	1	0	1
Mike	2	1	0

Real-Life Analogy

Encoding = Translating Languages

Imagine teaching a robot that only understands numbers. To tell it about colors, you must translate words like "Red" or "Blue" into numbers. Encoding is exactly that — translating human language into machine language.

When to Use Which Encoding?

Use Label Encoding for ordinal columns (Size, Rating).
Use One-Hot Encoding for nominal columns with few categories.
Use Ordinal Encoding for manually ordered categories.
Use Binary / Frequency / Target Encoding for high-cardinality columns.
Use Hashing for extremely large datasets.

Common Mistakes to Avoid

Mistake 1 Using Label Encoding on nominal data — leads to misleading order.

Mistake 2 Applying One-Hot Encoding on high-cardinality columns — explodes dimensionality.

Mistake 3 Performing Target Encoding on full dataset — causes data leakage.

Correct Approach Always encode after train-test split, and use the same encoder for both.

Best Practices

Quick Tips

Identify whether the column is nominal or ordinal.
Drop one column after One-Hot Encoding to avoid dummy variable trap (use drop_first=True).
Use pipelines with ColumnTransformer for clean preprocessing.
Save the fitted encoder for use on new/unseen data.
Handle unknown categories in test data using handle_unknown="ignore".

Conceptual View

RULE

Text Data + Encoding = Numeric Data

Importance of Categorical Encoding

Enables ML Training

Most algorithms require numeric input
Without encoding, models fail

Improves Accuracy

Right encoding boosts performance
Reduces bias

Helps Model Generalize

Captures meaningful patterns
Improves feature representation

Avoids Errors

Prevents crashes on string inputs
Cleaner pipeline

Key Takeaway

Encoding Categorical Data is the bridge between human-readable text and machine-readable numbers. Choosing the right technique — Label, One-Hot, Ordinal, Binary, or Target Encoding — depends on your dataset type, cardinality, and ML algorithm.