Table of Contents

Deliver an exploratory notebook and tested reusable data module

Rumman Ansari November 07, 2026 1 views Subject Details

INDEPENDENT LAB • 120 MIN

Deliver an Exploratory Notebook and Tested Reusable Data Module

Build a reproducible scientific workflow that separates exploration from reusable Python code, reruns from a clean notebook kernel, and verifies shapes, invalid inputs and deterministic preprocessing with automated tests.

Lab objective: Deliver an exploratory notebook, a reusable data module and an automated test suite. The notebook must run from a clean kernel without relying on hidden state. The tests must cover expected shapes, invalid inputs and deterministic preprocessing.

1. Independent Lab Scenario

A data science team has received a small customer dataset for exploration. The first analysis was completed directly in a notebook, but the notebook contains repeated cleaning code, variables created out of order and no automated verification. The workflow must be redesigned so that the notebook communicates the investigation while reusable, testable logic lives in a Python module.

Your submission should demonstrate that scientific exploration and software engineering can support each other:

DESIGN PRINCIPLE

Notebook = narrative + exploration + plots + conclusions
Data module = reusable transformations + validation
Tests = executable evidence of expected behavior

2. Acceptance Criteria

Criterion	Acceptance evidence
Clean-kernel execution	The notebook restarts and runs from first cell to last without manual intervention.
Reusable module	Cleaning, validation and feature preparation are implemented in an importable Python file.
Shape coverage	Tests verify raw, feature and transformed output shapes.
Invalid-input coverage	Tests reject missing columns, invalid targets, duplicate identifiers or infinite values.
Deterministic preprocessing	Repeated execution with the same input and configuration produces equivalent output.
Exploratory evidence	The notebook includes data inspection, missingness, distributions and documented conclusions.
Leakage protection	Any learned preprocessing is fitted on training data only.

3. Prerequisites

Required Knowledge and Tools

Basic Python functions, modules, exceptions and classes.
Basic NumPy and pandas operations.
Basic Matplotlib plotting.
Familiarity with Jupyter Notebook or JupyterLab.
Basic pytest fixtures and assertions.
Understanding of training and held-out datasets.

python -m pip install numpy pandas matplotlib scikit-learn pytest jupyter joblib nbconvert

4. Required Project Structure

exploratory_data_lab/
├── data/
│   └── customer_lab.csv
├── notebooks/
│   └── 01_customer_exploration.ipynb
├── src/
│   ├── __init__.py
│   └── data_utils.py
├── tests/
│   └── test_data_utils.py
├── artifacts/
├── create_sample_data.py
├── run_pipeline.py
├── requirements.txt
└── README.md

Path rule: The notebook must use stable project-relative paths. Do not depend on an absolute path from one computer such as C:\Users\Name\Desktop\....

5. Create the Practice Dataset

Create create_sample_data.py. This produces a small CSV with numerical, categorical and date fields, plus controlled missing and invalid values.

from pathlib import Path
import pandas as pd


OUTPUT_PATH = Path("data/customer_lab.csv")


def main() -> None:
    OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True)

    df = pd.DataFrame({
        "customer_id": list(range(2001, 2025)),
        "age": [
            23, 27, 31, None, 38, 42, 46, 51,
            29, 34, 37, 44, 49, 26, None, 33,
            36, 41, 55, 30, 28, 47, 40, 35
        ],
        "monthly_income": [
            30000, 35000, 42000, 47000, None, 63000,
            72000, 84000, 49000, 52000, 58000, "unknown",
            78000, 37000, 45000, 54000, 60000, 66000,
            92000, 46000, 39000, 75000, 62000, 55000
        ],
        "city": [
            "Kolkata", "Delhi", "Mumbai", " kolkata ", "Pune", "Delhi",
            "Mumbai", "Kolkata", None, "Pune", "Delhi", "Mumbai",
            "Kolkata", "Pune", "Delhi", "Mumbai", "Kolkata", "Delhi",
            "Pune", "Mumbai", "Kolkata", "Delhi", "Pune", "Bengaluru"
        ],
        "segment": [
            "Standard", "Standard", "Premium", "Standard", None, "Premium",
            "Premium", "Premium", "Standard", "Standard", "Premium", "Premium",
            "Premium", "Standard", "Standard", "Premium", "Standard", "Premium",
            "Premium", "Standard", "Standard", "Premium", "Premium", "Standard"
        ],
        "signup_date": [
            "2025-01-05", "2025-01-18", "2025-02-03", "2025-02-20",
            "2025-03-09", "2025-03-22", "invalid", "2025-04-15",
            "2025-05-01", "2025-05-19", "2025-06-02", "2025-06-18",
            "2025-07-04", None, "2025-08-01", "2025-08-17",
            "2025-09-05", "2025-09-21", "2025-10-03", "2025-10-19",
            "2025-11-02", "2025-11-16", "2025-12-01", "2025-12-18"
        ],
        "purchased": [
            0, 0, 1, 0, 0, 1, 1, 1,
            0, 0, 1, 1, 1, 0, 0, 1,
            0, 1, 1, 0, 0, 1, 1, 0
        ]
    })

    df.to_csv(OUTPUT_PATH, index=False)
    print(f"Saved {OUTPUT_PATH} with shape {df.shape}")


if __name__ == "__main__":
    main()

python create_sample_data.py

EXPECTED RAW SHAPE

\[ (24,7) \]

6. Implement the Reusable Data Module

Create src/data_utils.py. The module must not rely on variables defined by the notebook.

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from typing import Iterable

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler


REQUIRED_COLUMNS = {
    "customer_id",
    "age",
    "monthly_income",
    "city",
    "segment",
    "signup_date",
    "purchased",
}


@dataclass(frozen=True)
class DatasetSummary:
    rows: int
    columns: int
    missing_values: int
    duplicate_rows: int


def load_csv(path: str | Path) -> pd.DataFrame:
    """Load a non-empty CSV with unique column names."""
    csv_path = Path(path)

    if not csv_path.exists():
        raise FileNotFoundError(f"CSV not found: {csv_path.resolve()}")

    if csv_path.suffix.lower() != ".csv":
        raise ValueError("Expected a .csv file")

    df = pd.read_csv(csv_path)

    if df.empty:
        raise ValueError("Dataset is empty")

    if not df.columns.is_unique:
        raise ValueError("Duplicate column names detected")

    return df


def normalize_column_names(columns: Iterable[str]) -> list[str]:
    """Normalize column labels deterministically."""
    series = pd.Series(list(columns), dtype="string")
    return (
        series.str.strip()
        .str.lower()
        .str.replace(r"[^a-z0-9]+", "_", regex=True)
        .str.strip("_")
        .tolist()
    )


def clean_structure(df: pd.DataFrame) -> pd.DataFrame:
    """Apply deterministic parsing and text normalization."""
    cleaned = df.copy(deep=True)
    cleaned.columns = normalize_column_names(cleaned.columns)

    # Normalize recognized missing tokens case-insensitively.
    object_columns = cleaned.select_dtypes(include=["object", "string"]).columns
    for column in object_columns:
        values = cleaned[column].astype("string").str.strip()
        missing_mask = values.str.lower().isin(
            ["", "na", "n/a", "none", "null", "unknown", "?"]
        )
        cleaned[column] = values.mask(missing_mask, pd.NA)

    for column in ["age", "monthly_income"]:
        if column in cleaned.columns:
            cleaned[column] = pd.to_numeric(cleaned[column], errors="coerce")

    for column in ["city", "segment"]:
        if column in cleaned.columns:
            cleaned[column] = cleaned[column].astype("string").str.title()

    if "signup_date" in cleaned.columns:
        cleaned["signup_date"] = pd.to_datetime(
            cleaned["signup_date"],
            errors="coerce"
        )

    return cleaned


def validate_dataset(df: pd.DataFrame) -> None:
    """Validate the structural and business assumptions of the lab dataset."""
    missing_columns = REQUIRED_COLUMNS.difference(df.columns)
    if missing_columns:
        raise ValueError(f"Missing required columns: {sorted(missing_columns)}")

    if not df["customer_id"].is_unique:
        raise ValueError("customer_id must be unique")

    if df["purchased"].isna().any():
        raise ValueError("purchased contains missing values")

    target_values = set(df["purchased"].unique())
    if not target_values.issubset({0, 1}):
        raise ValueError(f"Unexpected purchased values: {sorted(target_values)}")

    if np.isinf(df[["age", "monthly_income"]].to_numpy(dtype=float)).any():
        raise ValueError("Infinite numerical values detected")

    valid_age = df["age"].dropna().between(18, 100)
    if not valid_age.all():
        raise ValueError("age contains values outside 18-100")


def summarize_dataset(df: pd.DataFrame) -> DatasetSummary:
    """Return a compact immutable summary."""
    return DatasetSummary(
        rows=df.shape[0],
        columns=df.shape[1],
        missing_values=int(df.isna().sum().sum()),
        duplicate_rows=int(df.duplicated().sum()),
    )


def missing_value_report(df: pd.DataFrame) -> pd.DataFrame:
    """Return dtype and missingness evidence by column."""
    return pd.DataFrame({
        "dtype": df.dtypes.astype(str),
        "missing_count": df.isna().sum(),
        "missing_percent": df.isna().mean().mul(100).round(2),
    }).sort_values("missing_percent", ascending=False)


def build_model_features(df: pd.DataFrame) -> pd.DataFrame:
    """Drop identifiers and derive deterministic calendar features."""
    features = df.copy(deep=True)
    features = features.drop(columns=["customer_id"], errors="ignore")

    if "signup_date" in features.columns:
        if not pd.api.types.is_datetime64_any_dtype(features["signup_date"]):
            raise TypeError("signup_date must be parsed before feature engineering")

        features["signup_year"] = features["signup_date"].dt.year
        features["signup_month"] = features["signup_date"].dt.month
        features["signup_day_of_week"] = features["signup_date"].dt.dayofweek
        features = features.drop(columns=["signup_date"])

    return features


def infer_feature_groups(df: pd.DataFrame) -> tuple[list[str], list[str]]:
    """Infer model input groups from training features."""
    numeric = df.select_dtypes(include=["number"]).columns.tolist()
    categorical = df.select_dtypes(
        include=["object", "string", "category", "bool"]
    ).columns.tolist()

    unsupported = [
        column for column in df.columns
        if column not in numeric and column not in categorical
    ]
    if unsupported:
        raise TypeError(f"Unsupported dtypes: {unsupported}")

    return numeric, categorical


def build_preprocessor(
    numeric_features: list[str],
    categorical_features: list[str]
) -> ColumnTransformer:
    """Return an unfitted, deterministic preprocessing graph."""
    numeric_pipeline = Pipeline(steps=[
        ("imputer", SimpleImputer(strategy="median", add_indicator=True)),
        ("scaler", StandardScaler()),
    ])

    categorical_pipeline = Pipeline(steps=[
        ("imputer", SimpleImputer(strategy="most_frequent")),
        (
            "encoder",
            OneHotEncoder(handle_unknown="ignore", sparse_output=False),
        ),
    ])

    return ColumnTransformer(
        transformers=[
            ("numeric", numeric_pipeline, numeric_features),
            ("categorical", categorical_pipeline, categorical_features),
        ],
        remainder="drop",
        verbose_feature_names_out=False,
    )


def to_processed_dataframe(
    preprocessor: ColumnTransformer,
    df: pd.DataFrame,
    *,
    fit: bool
) -> pd.DataFrame:
    """Fit optionally, transform and return a labeled DataFrame."""
    values = (
        preprocessor.fit_transform(df)
        if fit
        else preprocessor.transform(df)
    )

    return pd.DataFrame(
        values,
        columns=preprocessor.get_feature_names_out(),
        index=df.index,
    )

7. Design Assumptions

Assumption	Implementation decision
Each customer appears once	`customer_id` must be unique.
The target is binary	`purchased` accepts only 0 and 1.
Missing numeric values are valid but infinity is not	NaN is imputed later; infinity raises an error.
Identifiers are not model measurements	`customer_id` is excluded from model features.
Dates have predictable semantics	Year, month and day-of-week are derived deterministically.
Learned rules must not inspect held-out data	The preprocessor fits only on training features.

8. Implement the Automated Test Suite

Create tests/test_data_utils.py.

import numpy as np
import pandas as pd
import pytest

from src.data_utils import (
    build_model_features,
    build_preprocessor,
    clean_structure,
    infer_feature_groups,
    missing_value_report,
    normalize_column_names,
    summarize_dataset,
    to_processed_dataframe,
    validate_dataset,
)


@pytest.fixture
def valid_raw_df():
    return pd.DataFrame({
        "customer_id": [1, 2, 3, 4, 5, 6],
        "age": [22, None, 35, 41, 29, 50],
        "monthly_income": [30000, "unknown", 50000, 65000, 42000, 80000],
        "city": ["Kolkata", " delhi ", None, "Mumbai", "Pune", "Delhi"],
        "segment": ["Standard", "Premium", "Standard", None, "Standard", "Premium"],
        "signup_date": [
            "2025-01-01", "invalid", "2025-02-01",
            "2025-03-01", None, "2025-04-01"
        ],
        "purchased": [0, 1, 0, 1, 0, 1],
    })


def test_normalized_column_names_are_deterministic():
    source = [" Customer ID ", "Monthly Income", "Signup-Date"]

    first = normalize_column_names(source)
    second = normalize_column_names(source)

    assert first == ["customer_id", "monthly_income", "signup_date"]
    assert first == second


def test_clean_structure_preserves_shape(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    assert cleaned.shape == valid_raw_df.shape


def test_clean_structure_does_not_mutate_input(valid_raw_df):
    original = valid_raw_df.copy(deep=True)
    clean_structure(valid_raw_df)
    pd.testing.assert_frame_equal(valid_raw_df, original)


def test_clean_structure_is_deterministic(valid_raw_df):
    first = clean_structure(valid_raw_df)
    second = clean_structure(valid_raw_df)
    pd.testing.assert_frame_equal(first, second)


def test_clean_structure_parses_expected_types(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)

    assert pd.api.types.is_numeric_dtype(cleaned["age"])
    assert pd.api.types.is_numeric_dtype(cleaned["monthly_income"])
    assert pd.api.types.is_datetime64_any_dtype(cleaned["signup_date"])
    assert cleaned.loc[1, "city"] == "Delhi"
    assert pd.isna(cleaned.loc[1, "monthly_income"])
    assert pd.isna(cleaned.loc[1, "signup_date"])


def test_summary_has_expected_shape(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    summary = summarize_dataset(cleaned)

    assert summary.rows == 6
    assert summary.columns == 7
    assert summary.missing_values >= 1


def test_missing_report_covers_every_column(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    report = missing_value_report(cleaned)

    assert report.shape[0] == cleaned.shape[1]
    assert set(report.index) == set(cleaned.columns)
    assert {"dtype", "missing_count", "missing_percent"}.issubset(report.columns)


def test_validate_dataset_accepts_valid_clean_data(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    validate_dataset(cleaned)


def test_validate_dataset_rejects_missing_column(valid_raw_df):
    cleaned = clean_structure(valid_raw_df).drop(columns=["city"])

    with pytest.raises(ValueError, match="Missing required columns"):
        validate_dataset(cleaned)


def test_validate_dataset_rejects_duplicate_customer_id(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    cleaned.loc[1, "customer_id"] = cleaned.loc[0, "customer_id"]

    with pytest.raises(ValueError, match="customer_id must be unique"):
        validate_dataset(cleaned)


def test_validate_dataset_rejects_invalid_target(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    cleaned.loc[0, "purchased"] = 2

    with pytest.raises(ValueError, match="Unexpected purchased values"):
        validate_dataset(cleaned)


def test_validate_dataset_rejects_infinity(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    cleaned.loc[0, "monthly_income"] = np.inf

    with pytest.raises(ValueError, match="Infinite numerical values"):
        validate_dataset(cleaned)


def test_validate_dataset_rejects_invalid_age(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    cleaned.loc[0, "age"] = 150

    with pytest.raises(ValueError, match="outside 18-100"):
        validate_dataset(cleaned)


def test_feature_engineering_has_expected_shape(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    X = cleaned.drop(columns=["purchased"])
    features = build_model_features(X)

    # Six source feature columns: remove ID and replace date with three fields.
    assert X.shape == (6, 6)
    assert features.shape == (6, 7)
    assert "customer_id" not in features.columns
    assert "signup_date" not in features.columns


def test_feature_engineering_is_deterministic(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    X = cleaned.drop(columns=["purchased"])

    first = build_model_features(X)
    second = build_model_features(X)

    pd.testing.assert_frame_equal(first, second)


def test_feature_engineering_requires_parsed_date(valid_raw_df):
    raw_X = valid_raw_df.drop(columns=["purchased"])

    with pytest.raises(TypeError, match="signup_date must be parsed"):
        build_model_features(raw_X)


def test_train_and_holdout_have_matching_processed_schema(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    X = build_model_features(cleaned.drop(columns=["purchased"]))

    train = X.iloc[:4].copy()
    holdout = X.iloc[4:].copy()

    numeric, categorical = infer_feature_groups(train)
    preprocessor = build_preprocessor(numeric, categorical)

    train_result = to_processed_dataframe(preprocessor, train, fit=True)
    holdout_result = to_processed_dataframe(preprocessor, holdout, fit=False)

    assert train_result.shape[0] == 4
    assert holdout_result.shape[0] == 2
    assert list(train_result.columns) == list(holdout_result.columns)
    assert np.isfinite(train_result.to_numpy()).all()
    assert np.isfinite(holdout_result.to_numpy()).all()


def test_repeated_fitting_on_same_training_data_is_deterministic(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    X = build_model_features(cleaned.drop(columns=["purchased"]))

    numeric, categorical = infer_feature_groups(X)

    first_preprocessor = build_preprocessor(numeric, categorical)
    second_preprocessor = build_preprocessor(numeric, categorical)

    first = to_processed_dataframe(first_preprocessor, X, fit=True)
    second = to_processed_dataframe(second_preprocessor, X, fit=True)

    pd.testing.assert_frame_equal(first, second)

9. Run the Tests

pytest -q

For a more descriptive report:

pytest -v

Required evidence: Include the final test output in the lab submission. A successful clean-kernel notebook does not replace module tests, and passing tests do not replace a clean notebook rerun.

10. Build the Exploratory Notebook

Create notebooks/01_customer_exploration.ipynb. Use the following cell order. Each cell must depend only on previous cells.

Notebook Cell 1: Title and Objective

# Customer Data Exploration

## Objective
Inspect the customer dataset, validate data quality, visualize feature distributions,
create a reproducible training/held-out split, and demonstrate use of the reusable
`src.data_utils` module.

Notebook Cell 2: Stable Project Path and Imports

from pathlib import Path
import sys

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

PROJECT_ROOT = Path.cwd().resolve()
if PROJECT_ROOT.name == "notebooks":
    PROJECT_ROOT = PROJECT_ROOT.parent

if str(PROJECT_ROOT) not in sys.path:
    sys.path.insert(0, str(PROJECT_ROOT))

from src.data_utils import (
    build_model_features,
    build_preprocessor,
    clean_structure,
    infer_feature_groups,
    load_csv,
    missing_value_report,
    summarize_dataset,
    to_processed_dataframe,
    validate_dataset,
)

DATA_PATH = PROJECT_ROOT / "data" / "customer_lab.csv"
SEED = 42
TARGET = "purchased"

print("Project root:", PROJECT_ROOT)
print("Data path:", DATA_PATH)

Notebook Cell 3: Record Environment

import platform
import sklearn

print("Python:", platform.python_version())
print("NumPy:", np.__version__)
print("pandas:", pd.__version__)
print("scikit-learn:", sklearn.__version__)
print("Experiment seed:", SEED)

Notebook Cell 4: Load and Clean

raw_df = load_csv(DATA_PATH)
clean_df = clean_structure(raw_df)
validate_dataset(clean_df)

print("Raw shape:", raw_df.shape)
print("Clean shape:", clean_df.shape)
clean_df.head()

Notebook Cell 5: Dataset Summary

summary = summarize_dataset(clean_df)
summary

Notebook Cell 6: Dtypes and Missingness

display(clean_df.dtypes.to_frame("dtype"))
display(missing_value_report(clean_df))

Notebook Cell 7: Numerical Distributions

numeric_columns = ["age", "monthly_income"]

fig, axes = plt.subplots(1, 2, figsize=(12, 4))

for axis, column in zip(axes, numeric_columns):
    axis.hist(clean_df[column].dropna(), bins=10)
    axis.set_title(f"Distribution of {column}")
    axis.set_xlabel(column)
    axis.set_ylabel("Frequency")
    axis.grid(True)

fig.tight_layout()
plt.show()

Notebook Cell 8: Categorical Distributions

categorical_columns = ["city", "segment"]

fig, axes = plt.subplots(1, 2, figsize=(13, 4))

for axis, column in zip(axes, categorical_columns):
    counts = clean_df[column].fillna("Missing").value_counts()
    axis.bar(counts.index.astype(str), counts.values)
    axis.set_title(f"Counts for {column}")
    axis.set_xlabel(column)
    axis.set_ylabel("Count")
    axis.tick_params(axis="x", rotation=30)
    axis.grid(True, axis="y")

fig.tight_layout()
plt.show()

Notebook Cell 9: Missingness Visualization

missing_percent = clean_df.isna().mean().mul(100).sort_values(ascending=False)

plt.figure(figsize=(10, 4))
plt.bar(missing_percent.index, missing_percent.values)
plt.title("Missing Values by Column")
plt.xlabel("Column")
plt.ylabel("Missing values (%)")
plt.xticks(rotation=30)
plt.grid(True, axis="y")
plt.show()

Notebook Cell 10: Separate Target and Split

X = clean_df.drop(columns=[TARGET])
y = clean_df[TARGET].astype("int64")

X_train, X_holdout, y_train, y_holdout = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=SEED,
    stratify=y,
)

print("X:", X.shape)
print("y:", y.shape)
print("X_train:", X_train.shape)
print("X_holdout:", X_holdout.shape)
print("y_train:", y_train.shape)
print("y_holdout:", y_holdout.shape)

Notebook Cell 11: Feature Engineering

X_train_features = build_model_features(X_train)
X_holdout_features = build_model_features(X_holdout)

print("Training features:", X_train_features.shape)
print("Held-out features:", X_holdout_features.shape)
display(X_train_features.head())

Notebook Cell 12: Fit Train-Only Preprocessing

numeric_features, categorical_features = infer_feature_groups(
    X_train_features
)

preprocessor = build_preprocessor(
    numeric_features,
    categorical_features,
)

X_train_processed = to_processed_dataframe(
    preprocessor,
    X_train_features,
    fit=True,
)

X_holdout_processed = to_processed_dataframe(
    preprocessor,
    X_holdout_features,
    fit=False,
)

print("Processed training:", X_train_processed.shape)
print("Processed held-out:", X_holdout_processed.shape)

Notebook Cell 13: Validate Processed Outputs

assert X_train_processed.shape[0] == y_train.shape[0]
assert X_holdout_processed.shape[0] == y_holdout.shape[0]
assert list(X_train_processed.columns) == list(X_holdout_processed.columns)
assert np.isfinite(X_train_processed.to_numpy()).all()
assert np.isfinite(X_holdout_processed.to_numpy()).all()

print("Processed output validation passed.")

Notebook Cell 14: Conclusions

## Conclusions

- The raw and structurally cleaned datasets retain the same row and column counts.
- Missing values exist in selected numeric, categorical and date fields.
- Learned preprocessing was fitted only on the training subset.
- The held-out subset was transformed with the fitted training preprocessor.
- Training and held-out processed matrices have matching feature schemas.
- Reusable data logic is implemented outside the notebook and protected by tests.

11. Shape Reasoning

The complete dataset contains 24 rows and 7 columns. Separating the target produces:

SOURCE SHAPES

\[ (24,7) \rightarrow X:(24,6) + y:(24,) \]

A 25% held-out split produces:

SPLIT SHAPES

\[ X_{train}:(18,6),\quad X_{holdout}:(6,6) \] \[ y_{train}:(18,),\quad y_{holdout}:(6,) \]

Feature engineering removes one identifier column and replaces one date column with three calendar columns:

ENGINEERED SHAPES

\[ (18,6) - 1_{id} - 1_{date} + 3_{calendar} = (18,7) \] \[ (6,6) - 1_{id} - 1_{date} + 3_{calendar} = (6,7) \]

After fitted imputation, missing indicators and one-hot encoding:

MODEL MATRICES

\[ X_{train}^{processed}:(18,k) \] \[ X_{holdout}^{processed}:(6,k) \]

The same training-fitted preprocessor guarantees a common value of \(k\) and a common column order.

12. Deterministic Preprocessing

A deterministic transformation produces the same result for the same input and configuration. The functions in this lab avoid uncontrolled randomness. The train/held-out split uses an explicit seed.

DETERMINISM

\[ f(X,config)=Y \Rightarrow f(X,config)=Y \]

Determinism does not mean all scientific experiments are universally identical across every platform. It means that the workflow controls its known sources of variation and records the assumptions needed for reproduction.

Reproducibility Evidence

Fixed dataset file or version.
Fixed split seed.
Recorded package versions.
No notebook cells depending on future or hidden state.
Pure or non-mutating deterministic cleaning utilities.
Automated equality tests for repeated preprocessing.

13. Verify the Notebook from a Clean Kernel

Manual evidence:

Jupyter Verification

Open the notebook.
Select Kernel → Restart Kernel and Run All Cells.
Confirm every cell completes in order.
Confirm no variable is referenced before creation.
Confirm plots, shapes and conclusions appear.
Save the executed notebook.

Command-line evidence with nbconvert:

jupyter nbconvert \
  --to notebook \
  --execute notebooks/01_customer_exploration.ipynb \
  --output 01_customer_exploration.executed.ipynb \
  --ExecutePreprocessor.timeout=120

Acceptance rule: The executed notebook file should be created without an execution exception. A notebook that works only after manually running cells in a special order is not accepted.

14. Add a Simple Pipeline Runner

Create run_pipeline.py as non-notebook proof that the module is reusable.

from pathlib import Path

import joblib
import numpy as np
from sklearn.model_selection import train_test_split

from src.data_utils import (
    build_model_features,
    build_preprocessor,
    clean_structure,
    infer_feature_groups,
    load_csv,
    to_processed_dataframe,
    validate_dataset,
)


DATA_PATH = Path("data/customer_lab.csv")
ARTIFACT_PATH = Path("artifacts/preprocessor.joblib")
TARGET = "purchased"
SEED = 42


def main() -> None:
    raw_df = load_csv(DATA_PATH)
    clean_df = clean_structure(raw_df)
    validate_dataset(clean_df)

    X = clean_df.drop(columns=[TARGET])
    y = clean_df[TARGET].astype("int64")

    X_train, X_holdout, y_train, y_holdout = train_test_split(
        X,
        y,
        test_size=0.25,
        random_state=SEED,
        stratify=y,
    )

    X_train_features = build_model_features(X_train)
    X_holdout_features = build_model_features(X_holdout)

    numeric, categorical = infer_feature_groups(X_train_features)
    preprocessor = build_preprocessor(numeric, categorical)

    train_processed = to_processed_dataframe(
        preprocessor,
        X_train_features,
        fit=True,
    )
    holdout_processed = to_processed_dataframe(
        preprocessor,
        X_holdout_features,
        fit=False,
    )

    assert train_processed.shape[0] == y_train.shape[0]
    assert holdout_processed.shape[0] == y_holdout.shape[0]
    assert list(train_processed.columns) == list(holdout_processed.columns)
    assert np.isfinite(train_processed.to_numpy()).all()
    assert np.isfinite(holdout_processed.to_numpy()).all()

    ARTIFACT_PATH.parent.mkdir(parents=True, exist_ok=True)
    joblib.dump(preprocessor, ARTIFACT_PATH)

    print("Training shape:", train_processed.shape)
    print("Held-out shape:", holdout_processed.shape)
    print("Saved:", ARTIFACT_PATH)


if __name__ == "__main__":
    main()

python run_pipeline.py

15. Invalid Inputs That Must Be Tested

Invalid input	Expected behavior
Missing required column	Raise `ValueError` naming missing columns.
Duplicate customer identifier	Raise `ValueError`.
Target outside 0 and 1	Raise `ValueError` showing unexpected values.
Infinite numerical value	Raise `ValueError`.
Age outside documented range	Raise `ValueError`.
Unparsed date passed to feature engineering	Raise `TypeError`.
Empty or missing CSV	Raise a clear file or data error.

16. Notebook Quality Rules

Reject

Cells must be run in a hidden order.
Absolute machine-specific paths are required.
Reusable functions are copied throughout the notebook.
Outputs have no interpretation.
The held-out dataset influences fitted preprocessing.

Accept

Restart and Run All succeeds.
Paths are project-relative.
Reusable code is imported from the module.
Plots and tables have written conclusions.
Training-only fitting is explicit.

17. Required Lab Evidence

Submission Package

The executed exploratory notebook.
The reusable src/data_utils.py module.
The complete tests/test_data_utils.py suite.
Terminal output showing all tests passed.
Terminal output or executed notebook proving clean-kernel execution.
Dataset shape and missingness evidence.
Numerical and categorical distribution plots.
Train-only preprocessing code and held-out transformation code.
The saved fitted preprocessor.
A brief README showing installation and execution commands.

18. Suggested README

# Exploratory Data Lab

## Setup

```bash
python -m pip install -r requirements.txt
```

## Create the dataset

```bash
python create_sample_data.py
```

## Run tests

```bash
pytest -q
```

## Execute the reusable pipeline

```bash
python run_pipeline.py
```

## Execute the notebook from a clean kernel

```bash
jupyter nbconvert --to notebook --execute \
  notebooks/01_customer_exploration.ipynb \
  --output 01_customer_exploration.executed.ipynb \
  --ExecutePreprocessor.timeout=120
```

19. Self-Review Questions

Explain Your Design

Why is reusable cleaning code placed outside the notebook?
Which preprocessing steps are deterministic and which learn from training data?
Why does a clean-kernel rerun detect hidden notebook state?
Why is input mutation tested?
Why must training and held-out processed columns have identical order?
How does the deterministic preprocessing test differ from setting a random seed?
What invalid input produced the most useful validation test?

20. Final Acceptance Checklist

Before Submission

The project installs from documented dependencies.
The sample dataset can be recreated.
The notebook runs from a clean kernel.
The notebook uses only project-relative paths.
The notebook imports reusable logic from src/data_utils.py.
The notebook includes plots and written observations.
The tests cover required shapes.
The tests cover invalid inputs.
The tests prove deterministic cleaning and feature engineering.
The tests prove the original input is not mutated.
The preprocessor fits only on training data.
The held-out data uses only the fitted transform.
All pytest tests pass.
The processed matrices contain finite values and matching schemas.

Key Takeaway

A trustworthy scientific workflow separates exploration from reusable implementation. The notebook should explain the investigation and rerun from a clean kernel. The data module should expose deterministic, validated transformations. The test suite should prove shape contracts, reject invalid inputs and confirm reproducible preprocessing behavior.

Get Neural Networks and Deep Learning updates by email

Short, practical lessons and course updates from Rumman Ansari. No spam.