Table of Contents

Knowledge check and evidence review: Scientific workflow and debugging

Rumman Ansari November 07, 2026 1 views Subject Details

ASSESSMENT • 30 MIN

Knowledge Check and Evidence Review: Scientific Workflow and Debugging

Demonstrate that you can build, diagnose and verify a reproducible scientific Python workflow using notebooks, reusable modules, assertions, logging, visualization and automated tests.

Assessment goal: Submit a short written or oral knowledge check plus evidence from the chapter. Acceptance requires a notebook that reruns from a clean kernel and automated tests covering shapes, invalid inputs and deterministic preprocessing.

1. Assessment Overview

This assessment checks practical understanding of scientific workflow and debugging. You should be able to explain why exploratory notebooks and reusable Python modules have different responsibilities, how tests protect numerical and preprocessing code, and how to prove that an experiment can be reproduced without hidden notebook state.

ACCEPTANCE REQUIREMENTS

Clean-kernel notebook + Shape tests + Invalid-input tests + Deterministic preprocessing

2. Evidence You Must Submit

Required Chapter Evidence

An exploratory notebook that executes from the first cell to the last after a kernel restart.
A reusable Python data or preprocessing module imported by the notebook.
Automated tests verifying important input and output shapes.
Automated tests for invalid data and expected exceptions.
A deterministic preprocessing test showing equivalent input and configuration produce equivalent output.
Evidence that learned preprocessing is fitted on training data only.
A held-out dataset transformed using the already-fitted preprocessor.
At least one useful data or experiment visualization with a short interpretation.

3. Knowledge Check: Scientific Workflow

Question 1

Why should reusable preprocessing logic be moved out of an experiment notebook?

Expected answer Reusable logic is easier to import, test, version and reuse when it is contained in normal Python modules. The notebook can then focus on experiment configuration, exploration, plots, observations and conclusions rather than duplicating implementation code.

Question 2

What does a clean-kernel notebook rerun prove?

Expected answer It provides evidence that the notebook does not depend on variables or imports left in memory by an earlier execution order. Every required object must be created by cells that execute before it is used.

Question 3

Why is a notebook that works only after manually executing cells in a special order unreliable?

Expected answer The visible notebook no longer completely describes the computation. Hidden kernel state influences the result, so another user or a clean automated run may fail or produce different results.

4. Knowledge Check: Plots and Logging

Question 4

What is the difference between a diagnostic plot and runtime logging?

Tool	Main purpose
Diagnostic plot	Reveal distributions, trends, outliers, learning curves and relationships visually.
Logging	Record runtime events, configuration, shapes, warnings and failure context.

Question 5

Why is logger.info("shape=%s", X.shape) generally more useful in reusable code than an unexplained print(X)?

Expected answer Logging supports severity levels and contextual messages while avoiding unnecessarily printing potentially large datasets. Recording a shape often gives the debugging information needed without overwhelming the output.

5. Knowledge Check: Assertions and Validation

Question 6

What is an appropriate use of an assertion in scientific code?

assert X.ndim == 2
assert X.shape[0] == y.shape[0]
assert np.isfinite(X).all()

Expected answer Assertions are useful for internal invariants that should hold during development, such as dimensional contracts or expected relationships between objects.

Question 7

Why should validation of untrusted external input not depend only on assert?

Expected answer Python assertions can be disabled under optimization. Required input validation should therefore use explicit checks and exceptions such as ValueError, TypeError or FileNotFoundError.

6. Knowledge Check: Shape Debugging

Question 8

If X.shape == (64, 20) and W.shape == (20, 8), what is the shape of X @ W?

SHAPE DERIVATION

\[ (64,20)@(20,8)\rightarrow(64,8) \]

Question 9

Why would (64,20) @ (16,8) fail?

Expected answer Matrix multiplication requires the inner dimensions to agree. The left matrix has 20 features while the right matrix expects 16 inputs, so the mathematical contract is invalid.

Question 10

What should you do before randomly reshaping an array to fix a shape error?

Expected answer Write the semantic meaning of each axis and derive the expected input and output shapes. A reshape should represent the intended data structure, not merely suppress an exception.

7. Knowledge Check: Non-Finite Values

Question 11

What does np.isfinite(x) help detect?

Expected answer It identifies values that are finite and therefore helps locate NaN, positive infinity and negative infinity in numerical data.

Question 12

A model loss becomes NaN at epoch 18. What is the better debugging strategy?

DIAGNOSTIC SEARCH

Last finite stage → responsible operation → first non-finite stage

Inspect batches, model outputs, loss inputs and gradients near the transition rather than examining only the final NaN.

8. Knowledge Check: Random State and Determinism

Question 13

Does setting one NumPy seed automatically make an experiment using Python, scikit-learn and PyTorch completely reproducible?

Expected answer No. Randomness can come from several libraries and stages, including data splitting, initialization, shuffling and augmentation. The experiment must explicitly control and record each stochastic component that matters.

Question 14

What does deterministic preprocessing mean?

DETERMINISTIC TRANSFORMATION

\[ f(X,C)=Y \Rightarrow f(X,C)=Y \]

For the same input \(X\) and the same configuration \(C\), repeated execution should produce equivalent output \(Y\).

Question 15

Why is a deterministic preprocessing test useful even when the code does not intentionally use randomness?

Expected answer It verifies that results do not depend on hidden mutable state, unstable ordering, accidental input mutation or another uncontrolled side effect.

9. Evidence Review: Clean-Kernel Notebook

The strongest evidence is an automated notebook execution from a fresh kernel.

jupyter nbconvert \
  --to notebook \
  --execute notebooks/01_customer_exploration.ipynb \
  --output 01_customer_exploration.executed.ipynb \
  --ExecutePreprocessor.timeout=120

Acceptance evidence The executed notebook is generated without an execution exception, and all calculations and plots are produced by cells that run in order.

Not sufficient “The notebook works on my computer” without restarting the kernel and rerunning all cells.

10. Evidence Review: Shape Tests

Your test suite should verify explicit shape contracts instead of relying only on visual inspection.

def test_feature_engineering_shape(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    X = cleaned.drop(columns=["purchased"])

    features = build_model_features(X)

    assert X.shape == (6, 6)
    assert features.shape == (6, 7)


def test_processed_train_holdout_rows(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    X = build_model_features(cleaned.drop(columns=["purchased"]))

    train = X.iloc[:4].copy()
    holdout = X.iloc[4:].copy()

    numeric, categorical = infer_feature_groups(train)
    preprocessor = build_preprocessor(numeric, categorical)

    train_result = to_processed_dataframe(preprocessor, train, fit=True)
    holdout_result = to_processed_dataframe(preprocessor, holdout, fit=False)

    assert train_result.shape[0] == 4
    assert holdout_result.shape[0] == 2
    assert train_result.shape[1] == holdout_result.shape[1]

11. Evidence Review: Invalid-Input Tests

Good tests confirm both successful behavior and deliberate failure behavior.

import numpy as np
import pytest


def test_rejects_duplicate_identifier(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    cleaned.loc[1, "customer_id"] = cleaned.loc[0, "customer_id"]

    with pytest.raises(ValueError, match="customer_id must be unique"):
        validate_dataset(cleaned)


def test_rejects_infinite_input(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    cleaned.loc[0, "monthly_income"] = np.inf

    with pytest.raises(ValueError, match="Infinite numerical values"):
        validate_dataset(cleaned)

Invalid condition	Expected evidence
Missing required column	A test expects `ValueError`.
Duplicate business identifier	A test rejects duplicate IDs.
Invalid target value	A test rejects values outside the target contract.
Infinite numeric input	A test rejects positive or negative infinity.
Invalid dimensional contract	A test or assertion rejects the unexpected shape.

12. Evidence Review: Deterministic Preprocessing

def test_cleaning_is_deterministic(valid_raw_df):
    first = clean_structure(valid_raw_df)
    second = clean_structure(valid_raw_df)

    pd.testing.assert_frame_equal(first, second)


def test_feature_engineering_is_deterministic(valid_raw_df):
    cleaned = clean_structure(valid_raw_df)
    X = cleaned.drop(columns=["purchased"])

    first = build_model_features(X)
    second = build_model_features(X)

    pd.testing.assert_frame_equal(first, second)

Also Test Input Mutation

def test_cleaning_does_not_mutate_input(valid_raw_df):
    original = valid_raw_df.copy(deep=True)

    clean_structure(valid_raw_df)

    pd.testing.assert_frame_equal(valid_raw_df, original)

Why this matters: A function that silently modifies its input can make repeated notebook execution depend on previous calls. Avoiding unintended mutation helps keep preprocessing understandable and reproducible.

13. Evidence Review: Train-Only Fitted Preprocessing

The chapter's preprocessing evidence should show an explicit leakage boundary:

X_train_processed = to_processed_dataframe(
    preprocessor,
    X_train_features,
    fit=True,
)

X_holdout_processed = to_processed_dataframe(
    preprocessor,
    X_holdout_features,
    fit=False,
)

Reject Refit the imputer, scaler or category encoder using held-out observations.

Accept Learn preprocessing state using training data and reuse that exact state for the held-out dataset.

14. Practical Code Check

Complete the missing lines:

# Training and held-out features already exist.

numeric, categorical = ______________________________

preprocessor = build_preprocessor(numeric, categorical)

train_processed = to_processed_dataframe(
    preprocessor,
    X_train_features,
    fit=________
)

holdout_processed = to_processed_dataframe(
    preprocessor,
    X_holdout_features,
    fit=________
)

assert ______________________________________________
assert np.isfinite(train_processed.to_numpy()).all()
assert np.isfinite(holdout_processed.to_numpy()).all()

Reference Answer

numeric, categorical = infer_feature_groups(X_train_features)

preprocessor = build_preprocessor(numeric, categorical)

train_processed = to_processed_dataframe(
    preprocessor,
    X_train_features,
    fit=True
)

holdout_processed = to_processed_dataframe(
    preprocessor,
    X_holdout_features,
    fit=False
)

assert list(train_processed.columns) == list(holdout_processed.columns)
assert np.isfinite(train_processed.to_numpy()).all()
assert np.isfinite(holdout_processed.to_numpy()).all()

15. Written Submission Template

Notebook Reproducibility

I restarted the notebook kernel and ran all cells in order. The notebook completed without relying on variables from an earlier session. Data loading, plots, splitting, preprocessing and output validation were recreated from the notebook itself and imported modules.

Reusable Module

I placed reusable loading, validation, deterministic cleaning and feature-engineering logic in an importable Python module. The notebook imports these functions rather than duplicating the implementation.

Shape and Invalid-Input Tests

My tests verify expected DataFrame and transformed-matrix shapes. Additional tests verify that invalid schema, duplicate identifiers, non-finite values and invalid targets fail with explicit exceptions.

Deterministic Preprocessing

I execute deterministic cleaning and feature engineering repeatedly with the same input and compare the complete outputs. I also verify that preprocessing functions do not unintentionally mutate their source DataFrame.

Training and Held-Out Separation

Learned preprocessing is fitted on training data only. The held-out dataset uses the already-fitted transformer and does not contribute to learned imputation, scaling or encoding parameters.

16. Short Oral Assessment

Be Prepared to Explain

Why a notebook should rerun from a clean kernel.
Why reusable preprocessing belongs in an importable module.
The difference between an assertion and an automated test.
How you diagnose a matrix or broadcasting shape error.
How you locate the first NaN or infinity in a numerical pipeline.
Why a single seed does not describe an entire experiment.
How your tests prove deterministic preprocessing.
Why the held-out dataset must not fit preprocessing parameters.

17. Assessment Rubric

Criterion	Accepted	Needs Revision
Clean-kernel execution	Notebook executes completely after restart or through automated execution.	Manual cell ordering or hidden state is required.
Reusable module	Core transformations and validation are imported from a Python module.	The notebook contains duplicated or state-dependent reusable logic.
Shape tests	Important input, engineered and processed shapes are asserted.	Shapes are only printed or visually inspected.
Invalid-input tests	Failure cases verify explicit expected exceptions.	Only successful inputs are tested.
Determinism	Repeated preprocessing results are programmatically compared.	Determinism is claimed without a repeatability test.
No unintended mutation	Source input remains unchanged after deterministic utilities run.	Functions silently alter source data without documentation.
Held-out integrity	Learned preprocessing fits only training data.	Held-out data influences learned preprocessing state.
Evidence interpretation	Plots, logs or outputs include meaningful conclusions.	Outputs are shown without interpretation.

18. Final Evidence Checklist

Before Submission

I submitted the executed exploratory notebook.
I verified the notebook after restarting the kernel.
I can reproduce the notebook through a documented execution process.
I submitted the reusable Python data module.
I submitted the automated test suite.
My tests cover expected shapes.
My tests cover invalid inputs and expected exceptions.
My tests prove deterministic preprocessing behavior.
My tests check for unintended input mutation.
I showed training-only fitting and held-out-only transformation.
I verified transformed data contains expected finite values.
I showed at least one visualization and explained what it indicates.
I included the final pytest result as evidence.

Acceptance Standard

Your chapter evidence is complete when the notebook reruns from a clean kernel, reusable scientific logic is separated into a testable module, and the automated tests demonstrate shape correctness, invalid-input handling and deterministic preprocessing. The final workflow should be understandable and reproducible without depending on hidden interactive state.

Get Neural Networks and Deep Learning updates by email

Short, practical lessons and course updates from Rumman Ansari. No spam.