Table of Contents

Prepare a small tabular dataset with documented transformations

Rumman Ansari November 07, 2026 1 views Subject Details

INDEPENDENT LAB • 120 MIN

Prepare a Small Tabular Dataset with Documented Transformations

Complete an end-to-end data preparation lab using pandas and scikit-learn. Show dataset shapes, define a missing-value policy, fit preprocessing on training data only, and preserve a held-out dataset for unbiased evaluation.

Lab deliverable: Produce a reproducible tabular-data preparation workflow containing the raw-data inspection, documented transformations, train/held-out split, missing-value policy, train-only fitted preprocessing, transformed shapes, validation checks and saved preprocessing artifact.

1. Lab Scenario

You are preparing a small customer dataset for a binary classification problem. The target column purchased indicates whether a customer purchased a product. The raw dataset contains numerical fields, categorical fields, a date, missing values, inconsistent text and a record identifier.

Your task is not to train the final predictive model. Your task is to produce a clean and defensible modeling dataset while ensuring that the held-out data does not influence preprocessing decisions that learn statistics from data.

LAB WORKFLOW

Raw data → Inspect → Deterministic cleaning → Split → Fit on train → Transform held-out → Validate

2. Prerequisites

Required Knowledge and Tools

Basic Python syntax, functions, lists and dictionaries.
Basic pandas DataFrame operations and missing-value inspection.
Basic understanding of training and test datasets.
Python with NumPy, pandas, scikit-learn and joblib installed.
A Python environment such as Jupyter Notebook, VS Code or PyCharm.

python -m pip install numpy pandas scikit-learn joblib

3. Lab Completion Criteria

Your submission should demonstrate all of the following:

Requirement	Evidence to show
Raw dataset inspection	Shape, sample rows, dtypes, missing counts and duplicates.
Documented transformations	A transformation log explaining what changed and why.
Missing-value policy	Explicit rules for numeric, categorical and date-derived fields.
Held-out dataset	A split created before fitting statistical preprocessing.
Train-only fit	`fit_transform()` on training data and only `transform()` on held-out data.
Shape evidence	Shapes before and after splitting and preprocessing.
Validation	Checks for schema consistency, missing values and row counts.

4. Create the Small Raw Dataset

Run the following code to create a reproducible CSV for the lab. It intentionally contains missing data, inconsistent category formatting and an invalid numerical token.

from pathlib import Path
import pandas as pd

raw_data = pd.DataFrame({
    "customer_id": [
        1001, 1002, 1003, 1004, 1005,
        1006, 1007, 1008, 1009, 1010,
        1011, 1012, 1013, 1014, 1015,
        1016, 1017, 1018, 1019, 1020
    ],
    "age": [
        24, 41, None, 36, 29,
        52, 44, 31, 27, 39,
        48, None, 33, 56, 26,
        45, 38, 30, 50, 35
    ],
    "monthly_income": [
        32000, 62000, 47000, None, 41000,
        79000, "unknown", 53000, 36000, 58000,
        71000, 49000, None, 85000, 39000,
        67000, 55000, 44000, 76000, 51000
    ],
    "city": [
        "Kolkata", "Delhi", "Kolkata", "Mumbai", " kolkata ",
        "Delhi", "Pune", None, "Mumbai", "Delhi",
        "Pune", "Kolkata", "Mumbai", "Delhi", "Pune",
        "Kolkata", "Mumbai", "Pune", "Delhi", "Bengaluru"
    ],
    "segment": [
        "Standard", "Premium", "Standard", "Standard", None,
        "Premium", "Premium", "Standard", "Standard", "Premium",
        "Premium", "Standard", "Standard", "Premium", "Standard",
        "Premium", None, "Standard", "Premium", "Standard"
    ],
    "signup_date": [
        "2025-01-10", "2025-01-15", "2025-02-01", "2025-02-18", "invalid",
        "2025-03-05", "2025-03-20", "2025-04-02", "2025-04-19", "2025-05-01",
        "2025-05-22", None, "2025-06-14", "2025-07-01", "2025-07-20",
        "2025-08-04", "2025-08-18", "2025-09-03", "2025-09-21", "2025-10-05"
    ],
    "purchased": [
        0, 1, 0, 0, 1,
        1, 1, 0, 0, 1,
        1, 0, 0, 1, 0,
        1, 0, 0, 1, 0
    ]
})

csv_path = Path("customer_lab_data.csv")
raw_data.to_csv(csv_path, index=False)

print(f"Saved: {csv_path.resolve()}")
print("Raw shape:", raw_data.shape)

The expected raw shape is:

RAW SHAPE

20 rows × 7 columns

5. Load and Inspect the Dataset

import pandas as pd


df = pd.read_csv("customer_lab_data.csv")

print("Shape:", df.shape)
print("\nFirst five rows:")
print(df.head())

print("\nData types:")
print(df.dtypes)

print("\nMissing values:")
print(df.isna().sum())

print("\nMissing percentages:")
print(df.isna().mean().mul(100).round(1))

print("\nExact duplicate rows:")
print(df.duplicated().sum())

print("\nSummary:")
print(df.describe(include="all").transpose())

Inspection Questions

Why might monthly_income be inferred as text instead of numeric?
Which columns contain missing values?
Is customer_id a measurement or an identifier?
Should signup_date remain raw text for modeling?
Are category values consistently formatted?

6. Document the Transformation Policy

Before changing the data, document the intended transformation for each field. This creates an auditable data-preparation design rather than an undocumented sequence of edits.

Column	Raw issue	Transformation	Reason
`customer_id`	Identifier	Exclude from model features	Its numeric magnitude has no intended predictive meaning.
`age`	Missing numeric values	Median imputation fitted on training data	Provides a robust baseline while preventing held-out leakage.
`monthly_income`	Missing values and `unknown`	Coerce invalid token to missing, then train-median imputation	Restore numerical meaning without using held-out statistics.
`city`	Whitespace, capitalization and missing category	Strip/title-case, then training most-frequent imputation and one-hot encoding	Unify categories and produce model-ready numerical features.
`segment`	Missing category	Training most-frequent imputation and one-hot encoding	Use a documented categorical baseline.
`signup_date`	Invalid or missing dates	Parse with invalid values as missing; derive year/month/day-of-week	Convert a date into structured features.
`purchased`	Target	Keep separate from feature preprocessing	Never expose the target as an input feature.

7. Missing-Value Policy

The lab uses an explicit missing-value policy. The important point is not that these policies are universally optimal. The important point is that they are declared, reproducible and fitted correctly.

NUMERIC POLICY

Replace missing numeric values with the training-set median and add a missingness indicator.

CATEGORICAL POLICY

Replace missing categories with the most frequent training category, then one-hot encode the training vocabulary.

DATE POLICY

Invalid dates become missing; year, month and day-of-week are derived and subsequently handled as numerical features.

Leakage rule: Deterministic parsing can be applied consistently, but statistics such as medians, most-frequent categories and scaling parameters must be learned from the training subset only.

8. Deterministic Cleaning

import numpy as np
import pandas as pd


def clean_structure(df: pd.DataFrame) -> pd.DataFrame:
    """Apply deterministic transformations without learning statistics."""
    cleaned = df.copy()

    cleaned.columns = (
        cleaned.columns
        .str.strip()
        .str.lower()
        .str.replace(r"[^a-z0-9]+", "_", regex=True)
        .str.strip("_")
    )

    # Convert known missing tokens to a common missing representation.
    cleaned = cleaned.replace(
        ["", " ", "unknown", "Unknown", "N/A", "NA", "null", "None", "?"],
        np.nan
    )

    cleaned["age"] = pd.to_numeric(cleaned["age"], errors="coerce")
    cleaned["monthly_income"] = pd.to_numeric(
        cleaned["monthly_income"], errors="coerce"
    )

    cleaned["city"] = cleaned["city"].astype("string").str.strip().str.title()
    cleaned["segment"] = cleaned["segment"].astype("string").str.strip().str.title()

    cleaned["signup_date"] = pd.to_datetime(
        cleaned["signup_date"], errors="coerce"
    )

    return cleaned


clean_df = clean_structure(df)

print("Shape after deterministic cleaning:", clean_df.shape)
print(clean_df.dtypes)
print(clean_df.isna().sum())

The number of rows and columns should remain unchanged because this stage changes representations rather than deleting observations:

EXPECTED SHAPE

\[ (20,7) \rightarrow (20,7) \]

9. Validate Business-Level Assumptions

def validate_raw_modeling_data(df: pd.DataFrame) -> None:
    required = {
        "customer_id",
        "age",
        "monthly_income",
        "city",
        "segment",
        "signup_date",
        "purchased"
    }

    missing_columns = required.difference(df.columns)
    if missing_columns:
        raise ValueError(f"Missing required columns: {sorted(missing_columns)}")

    if not df["customer_id"].is_unique:
        raise ValueError("customer_id must be unique for this lab dataset.")

    if df["purchased"].isna().any():
        raise ValueError("Target contains missing values.")

    if not set(df["purchased"].unique()).issubset({0, 1}):
        raise ValueError("purchased must contain only 0 and 1.")

    valid_age = df["age"].dropna().between(18, 100)
    if not valid_age.all():
        raise ValueError("Unexpected age detected.")


validate_raw_modeling_data(clean_df)
print("Raw modeling-data validation passed.")

10. Separate Features and Target

TARGET = "purchased"

X = clean_df.drop(columns=[TARGET])
y = clean_df[TARGET].astype("int64")

print("X shape:", X.shape)  # 20 rows, 6 source feature columns
print("y shape:", y.shape)  # 20 target values

SHAPE TRACE

\[ Data:(20,7) \rightarrow X:(20,6) + y:(20,) \]

11. Create a Held-Out Dataset Before Fitting Preprocessing

Use 25% of the records as a held-out test set. It will remain isolated from statistical fitting. The fixed random state makes the split reproducible.

from sklearn.model_selection import train_test_split

X_train, X_holdout, y_train, y_holdout = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42,
    stratify=y
)

print("X_train shape:", X_train.shape)
print("X_holdout shape:", X_holdout.shape)
print("y_train shape:", y_train.shape)
print("y_holdout shape:", y_holdout.shape)

With 20 records and a 25% held-out allocation, the expected split contains 15 training records and 5 held-out records.

EXPECTED SPLIT

\[ X_{train}:(15,6),\quad X_{holdout}:(5,6) \] \[ y_{train}:(15,),\quad y_{holdout}:(5,) \]

Do not do this Calculate the full dataset median, fill missing values, encode all categories and only then create the held-out set.

Correct design Create the held-out set first, fit statistical preprocessing with X_train, and only transform X_holdout afterward.

12. Preserve the Held-Out Dataset

Save a human-readable copy of the held-out records before statistical transformation. This makes the separation concrete and produces useful lab evidence.

holdout_snapshot = X_holdout.copy()
holdout_snapshot["purchased"] = y_holdout
holdout_snapshot.to_csv("held_out_raw.csv", index=False)

print("Held-out raw dataset saved.")
print(holdout_snapshot)

13. Build Deterministic Model Features

The identifier is removed. The parsed date is converted into year, month and day-of-week. These transformations do not calculate population statistics.

def build_model_features(df: pd.DataFrame) -> pd.DataFrame:
    features = df.copy()

    features = features.drop(columns=["customer_id"], errors="ignore")

    features["signup_year"] = features["signup_date"].dt.year
    features["signup_month"] = features["signup_date"].dt.month
    features["signup_day_of_week"] = features["signup_date"].dt.dayofweek
    features = features.drop(columns=["signup_date"])

    return features


X_train_features = build_model_features(X_train)
X_holdout_features = build_model_features(X_holdout)

print("Training features shape:", X_train_features.shape)
print("Held-out features shape:", X_holdout_features.shape)
print(X_train_features.dtypes)

The original six feature columns contain one identifier and one date. Dropping the identifier and replacing the date with three derived fields results in seven model-input columns before encoding:

FEATURE SHAPE

\[ (15,6) \rightarrow (15,7) \] \[ (5,6) \rightarrow (5,7) \]

14. Inspect Training Missingness Only

train_missing_report = pd.DataFrame({
    "dtype": X_train_features.dtypes.astype(str),
    "missing_count": X_train_features.isna().sum(),
    "missing_percent": X_train_features.isna().mean().mul(100).round(1)
}).sort_values("missing_percent", ascending=False)

print(train_missing_report)

This report describes the training subset from which imputation values will be learned. You may inspect held-out missingness for diagnostics, but you must not use it to choose fitted replacement values during the lab evaluation workflow.

15. Identify Numeric and Categorical Features

numeric_features = X_train_features.select_dtypes(
    include=["number"]
).columns.tolist()

categorical_features = X_train_features.select_dtypes(
    include=["object", "string", "category", "bool"]
).columns.tolist()

print("Numeric:", numeric_features)
print("Categorical:", categorical_features)

16. Build the Train-Only Preprocessor

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


numeric_pipeline = Pipeline(steps=[
    (
        "imputer",
        SimpleImputer(strategy="median", add_indicator=True)
    ),
    (
        "scaler",
        StandardScaler()
    )
])

categorical_pipeline = Pipeline(steps=[
    (
        "imputer",
        SimpleImputer(strategy="most_frequent")
    ),
    (
        "encoder",
        OneHotEncoder(
            handle_unknown="ignore",
            sparse_output=False
        )
    )
])

preprocessor = ColumnTransformer(
    transformers=[
        ("numeric", numeric_pipeline, numeric_features),
        ("categorical", categorical_pipeline, categorical_features)
    ],
    remainder="drop",
    verbose_feature_names_out=False
)

17. Fit on Training Data Only

# Training data is allowed to fit preprocessing parameters.
X_train_array = preprocessor.fit_transform(X_train_features)

# Held-out data must use the already-fitted preprocessing object.
X_holdout_array = preprocessor.transform(X_holdout_features)

feature_names = preprocessor.get_feature_names_out()

X_train_processed = pd.DataFrame(
    X_train_array,
    columns=feature_names,
    index=X_train_features.index
)

X_holdout_processed = pd.DataFrame(
    X_holdout_array,
    columns=feature_names,
    index=X_holdout_features.index
)

print("Processed training shape:", X_train_processed.shape)
print("Processed held-out shape:", X_holdout_processed.shape)

Evidence of isolation: There is no call to fit() or fit_transform() using X_holdout_features. The held-out set receives only transform().

18. Explain Why the Number of Columns Changes

The processed number of columns may be greater than seven because categorical variables are one-hot encoded and missing indicators may be added for numeric variables that contained missing values during fitting.

GENERAL SHAPE

\[ X_{train}^{raw}:(15,7) \rightarrow X_{train}^{processed}:(15,k) \] \[ X_{holdout}^{raw}:(5,7) \rightarrow X_{holdout}^{processed}:(5,k) \]

The exact value of \(k\) depends on the category levels learned from training data and the numeric columns for which the fitted imputer creates missing indicators. The important invariant is that both transformed datasets must have the same \(k\) columns in the same order.

19. Inspect the Values Learned from Training Data

numeric_imputer = (
    preprocessor
    .named_transformers_["numeric"]
    .named_steps["imputer"]
)

categorical_imputer = (
    preprocessor
    .named_transformers_["categorical"]
    .named_steps["imputer"]
)

encoder = (
    preprocessor
    .named_transformers_["categorical"]
    .named_steps["encoder"]
)

print("Numeric source columns:")
print(numeric_features)

print("\nTraining-fitted numeric replacement values:")
print(numeric_imputer.statistics_)

print("\nCategorical source columns:")
print(categorical_features)

print("\nTraining-fitted categorical replacement values:")
print(categorical_imputer.statistics_)

print("\nTraining-fitted category vocabularies:")
for column, categories in zip(categorical_features, encoder.categories_):
    print(column, "->", categories)

20. Mathematical Reason for Train-Only Fitting

If a numeric feature is imputed using its median, the replacement value must be calculated from training observations only.

TRAINING MEDIAN

\[ \tilde{x}_{train}=\operatorname{median}\{x_i : i \in train,\; x_i\;\text{observed}\} \]

The held-out set is transformed using this fixed value:

HELD-OUT IMPUTATION

\[ x_i^{*}=\tilde{x}_{train}\quad \text{when held-out }x_i\text{ is missing} \]

A scaler follows the same principle. If standardization is used:

TRAIN-ONLY SCALING

\[ z=\frac{x-\mu_{train}}{\sigma_{train}} \]

Neither \(\mu_{train}\) nor \(\sigma_{train}\) may contain information from the held-out observations.

21. Validate Processed Shapes and Values

import numpy as np


assert X_train_processed.shape[0] == y_train.shape[0]
assert X_holdout_processed.shape[0] == y_holdout.shape[0]

assert X_train_processed.shape[1] == X_holdout_processed.shape[1]
assert list(X_train_processed.columns) == list(X_holdout_processed.columns)

assert not X_train_processed.isna().any().any()
assert not X_holdout_processed.isna().any().any()

assert np.isfinite(X_train_processed.to_numpy()).all()
assert np.isfinite(X_holdout_processed.to_numpy()).all()

print("All processed-data checks passed.")

22. Save Lab Outputs

Save the processed training and held-out datasets separately. Retain the target beside each exported dataset for lab inspection. In a production machine-learning workflow you may instead keep arrays and labels as separate artifacts.

train_output = X_train_processed.copy()
train_output["purchased"] = y_train.to_numpy()

holdout_output = X_holdout_processed.copy()
holdout_output["purchased"] = y_holdout.to_numpy()

train_output.to_csv("training_prepared.csv", index=False)
holdout_output.to_csv("held_out_prepared.csv", index=False)

print("Saved training_prepared.csv:", train_output.shape)
print("Saved held_out_prepared.csv:", holdout_output.shape)

23. Save the Fitted Preprocessor

import joblib

joblib.dump(preprocessor, "tabular_preprocessor.joblib")
print("Saved tabular_preprocessor.joblib")

The saved object contains the training-fitted imputation, scaling and encoding state. Future records should use this fitted object rather than fitting a new preprocessor.

24. Build a Transformation Log

Documenting transformations is part of the lab. The following example creates a simple machine-readable transformation log.

transformation_log = pd.DataFrame([
    {
        "step": 1,
        "field": "all columns",
        "transformation": "Normalize column names",
        "fit_scope": "deterministic",
        "reason": "Consistent schema"
    },
    {
        "step": 2,
        "field": "monthly_income",
        "transformation": "Convert invalid numeric text to missing",
        "fit_scope": "deterministic",
        "reason": "Restore numeric dtype"
    },
    {
        "step": 3,
        "field": "city, segment",
        "transformation": "Trim and title-case text",
        "fit_scope": "deterministic",
        "reason": "Normalize labels"
    },
    {
        "step": 4,
        "field": "signup_date",
        "transformation": "Parse date and derive calendar features",
        "fit_scope": "deterministic",
        "reason": "Create model-ready date features"
    },
    {
        "step": 5,
        "field": "numeric features",
        "transformation": "Median imputation, missing indicators, scaling",
        "fit_scope": "training only",
        "reason": "Handle missingness and numerical scale"
    },
    {
        "step": 6,
        "field": "categorical features",
        "transformation": "Most-frequent imputation and one-hot encoding",
        "fit_scope": "training only",
        "reason": "Create numerical categorical representation"
    }
])

transformation_log.to_csv("transformation_log.csv", index=False)
print(transformation_log)

25. Produce a Shape Report

shape_report = pd.DataFrame([
    {"stage": "Raw complete dataset", "shape": str(df.shape)},
    {"stage": "Clean complete dataset", "shape": str(clean_df.shape)},
    {"stage": "Feature matrix X", "shape": str(X.shape)},
    {"stage": "Target y", "shape": str(y.shape)},
    {"stage": "Training before feature engineering", "shape": str(X_train.shape)},
    {"stage": "Held-out before feature engineering", "shape": str(X_holdout.shape)},
    {"stage": "Training model features", "shape": str(X_train_features.shape)},
    {"stage": "Held-out model features", "shape": str(X_holdout_features.shape)},
    {"stage": "Training processed", "shape": str(X_train_processed.shape)},
    {"stage": "Held-out processed", "shape": str(X_holdout_processed.shape)}
])

shape_report.to_csv("shape_report.csv", index=False)
print(shape_report.to_string(index=False))

26. Complete Independent Lab Script

The following version assembles the main lab workflow into one runnable script after customer_lab_data.csv has been created.

from pathlib import Path

import joblib
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


CSV_PATH = Path("customer_lab_data.csv")
TARGET = "purchased"
TEST_SIZE = 0.25
RANDOM_STATE = 42


def load_data(path: Path) -> pd.DataFrame:
    if not path.exists():
        raise FileNotFoundError(path)

    df = pd.read_csv(path)

    if df.empty:
        raise ValueError("Dataset is empty.")

    if not df.columns.is_unique:
        raise ValueError("Duplicate column names detected.")

    return df


def clean_structure(df: pd.DataFrame) -> pd.DataFrame:
    cleaned = df.copy()

    cleaned.columns = (
        cleaned.columns
        .str.strip()
        .str.lower()
        .str.replace(r"[^a-z0-9]+", "_", regex=True)
        .str.strip("_")
    )

    cleaned = cleaned.replace(
        ["", " ", "unknown", "Unknown", "N/A", "NA", "null", "None", "?"],
        np.nan
    )

    cleaned["age"] = pd.to_numeric(cleaned["age"], errors="coerce")
    cleaned["monthly_income"] = pd.to_numeric(
        cleaned["monthly_income"], errors="coerce"
    )
    cleaned["city"] = cleaned["city"].astype("string").str.strip().str.title()
    cleaned["segment"] = cleaned["segment"].astype("string").str.strip().str.title()
    cleaned["signup_date"] = pd.to_datetime(
        cleaned["signup_date"], errors="coerce"
    )

    return cleaned


def build_model_features(df: pd.DataFrame) -> pd.DataFrame:
    features = df.copy()
    features = features.drop(columns=["customer_id"], errors="ignore")

    features["signup_year"] = features["signup_date"].dt.year
    features["signup_month"] = features["signup_date"].dt.month
    features["signup_day_of_week"] = features["signup_date"].dt.dayofweek
    features = features.drop(columns=["signup_date"])

    return features


def main() -> None:
    raw_df = load_data(CSV_PATH)
    clean_df = clean_structure(raw_df)

    print("Raw shape:", raw_df.shape)
    print("Clean shape:", clean_df.shape)
    print("Missing values after structural cleaning:")
    print(clean_df.isna().sum())

    if TARGET not in clean_df.columns:
        raise ValueError(f"Missing target: {TARGET}")

    if clean_df[TARGET].isna().any():
        raise ValueError("Target contains missing values.")

    X = clean_df.drop(columns=[TARGET])
    y = clean_df[TARGET].astype("int64")

    X_train, X_holdout, y_train, y_holdout = train_test_split(
        X,
        y,
        test_size=TEST_SIZE,
        random_state=RANDOM_STATE,
        stratify=y
    )

    # Save untouched held-out feature/target snapshot for audit evidence.
    holdout_snapshot = X_holdout.copy()
    holdout_snapshot[TARGET] = y_holdout
    holdout_snapshot.to_csv("held_out_raw.csv", index=False)

    X_train_features = build_model_features(X_train)
    X_holdout_features = build_model_features(X_holdout)

    numeric_features = X_train_features.select_dtypes(
        include=["number"]
    ).columns.tolist()

    categorical_features = X_train_features.select_dtypes(
        include=["object", "string", "category", "bool"]
    ).columns.tolist()

    numeric_pipeline = Pipeline(steps=[
        ("imputer", SimpleImputer(strategy="median", add_indicator=True)),
        ("scaler", StandardScaler())
    ])

    categorical_pipeline = Pipeline(steps=[
        ("imputer", SimpleImputer(strategy="most_frequent")),
        (
            "encoder",
            OneHotEncoder(handle_unknown="ignore", sparse_output=False)
        )
    ])

    preprocessor = ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features)
        ],
        remainder="drop",
        verbose_feature_names_out=False
    )

    # Critical leakage boundary.
    train_array = preprocessor.fit_transform(X_train_features)
    holdout_array = preprocessor.transform(X_holdout_features)

    feature_names = preprocessor.get_feature_names_out()

    train_processed = pd.DataFrame(
        train_array,
        columns=feature_names,
        index=X_train_features.index
    )

    holdout_processed = pd.DataFrame(
        holdout_array,
        columns=feature_names,
        index=X_holdout_features.index
    )

    # Validation.
    assert train_processed.shape[0] == y_train.shape[0]
    assert holdout_processed.shape[0] == y_holdout.shape[0]
    assert list(train_processed.columns) == list(holdout_processed.columns)
    assert not train_processed.isna().any().any()
    assert not holdout_processed.isna().any().any()
    assert np.isfinite(train_processed.to_numpy()).all()
    assert np.isfinite(holdout_processed.to_numpy()).all()

    # Save processed datasets.
    train_output = train_processed.copy()
    train_output[TARGET] = y_train.to_numpy()
    train_output.to_csv("training_prepared.csv", index=False)

    holdout_output = holdout_processed.copy()
    holdout_output[TARGET] = y_holdout.to_numpy()
    holdout_output.to_csv("held_out_prepared.csv", index=False)

    # Save the fitted train-only preprocessor.
    joblib.dump(preprocessor, "tabular_preprocessor.joblib")

    # Save shape evidence.
    shape_report = pd.DataFrame([
        {"stage": "raw", "shape": str(raw_df.shape)},
        {"stage": "clean", "shape": str(clean_df.shape)},
        {"stage": "X_train", "shape": str(X_train.shape)},
        {"stage": "X_holdout", "shape": str(X_holdout.shape)},
        {"stage": "X_train_features", "shape": str(X_train_features.shape)},
        {"stage": "X_holdout_features", "shape": str(X_holdout_features.shape)},
        {"stage": "train_processed", "shape": str(train_processed.shape)},
        {"stage": "holdout_processed", "shape": str(holdout_processed.shape)}
    ])
    shape_report.to_csv("shape_report.csv", index=False)

    print("\n--- FINAL SHAPES ---")
    print(shape_report.to_string(index=False))
    print("\nTrain-only preprocessing completed successfully.")


if __name__ == "__main__":
    main()

27. Required Lab Evidence

Include These in Your Submission

The raw dataset shape and column dtypes.
A missing-value count or percentage report.
Your written missing-value policy.
The documented transformation table or transformation_log.csv.
The shapes of X_train and X_holdout.
The call showing fit_transform(X_train_features).
The call showing only transform(X_holdout_features) for held-out data.
The final processed training and held-out shapes.
Validation proving both processed datasets have the same columns.
The saved fitted preprocessing artifact.

28. Interpretation Questions

Answer these questions in your own words after running the lab.

Analysis Questions

Why was customer_id removed from the model features?
Which transformations were deterministic and which learned information from training data?
Why must the held-out set not contribute to median imputation?
Why can one-hot encoding increase the number of columns?
What happens when the held-out set contains a city that did not appear in training?
Why must training and held-out transformed columns have identical ordering?
When would a chronological split be more appropriate than this random stratified split?

29. Common Failure Modes

Failure	Correction
Impute the complete dataset before splitting	Split first; fit the imputer on training data only.
Call `fit_transform()` on held-out data	Call `transform()` using the training-fitted preprocessor.
Treat an identifier as continuous data	Remove it unless there is a justified feature representation.
Ignore invalid strings in numeric columns	Coerce known invalid values to missing and include them in the missing-value policy.
Manually create different dummy columns for train and held-out sets	Use a single training-fitted `OneHotEncoder`.
Report only final data and hide transformations	Save a transformation log and shape report.

30. Optional Extension

After the core lab is complete, you can add a simple classifier. Keep preprocessing and the model in a single scikit-learn Pipeline so the same leakage-safe boundary is maintained.

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

model = Pipeline(steps=[
    ("preprocessor", preprocessor),
    ("classifier", LogisticRegression(max_iter=1000))
])

# If using this extension, create a fresh unfitted preprocessor before fitting
# the complete model pipeline.
# model.fit(X_train_features, y_train)
# predictions = model.predict(X_holdout_features)

31. Final Lab Checklist

Self-Assessment

I showed the raw dataset shape.
I inspected and documented data types.
I measured missing values.
I documented every transformation and its purpose.
I defined an explicit missing-value policy.
I separated the target from the features.
I created a held-out dataset before fitting statistical preprocessing.
I fitted medians, category replacements, scaling and encoding from training data only.
I used only transform() on held-out data.
I showed intermediate and final shapes.
I verified that processed training and held-out schemas match.
I saved the fitted preprocessing object and documented outputs.

Key Takeaway

A strong tabular-data preparation workflow is reproducible and leakage-safe. Document transformations, preserve shapes, define missing-value rules, isolate a held-out dataset before statistical fitting, fit preprocessing on training data only, and reuse the fitted transformer everywhere else. The final dataset is valuable not only because it is clean, but because you can explain exactly how it was produced.

Get Neural Networks and Deep Learning updates by email

Short, practical lessons and course updates from Rumman Ansari. No spam.