Table of Contents

Knowledge check and evidence review: NumPy, pandas and data preparation

Rumman Ansari November 07, 2026 2 views Subject Details

ASSESSMENT • 30 MIN

Knowledge Check and Evidence Review: NumPy, pandas and Data Preparation

Demonstrate practical understanding of array shapes, pandas data preparation, missing-value handling, data leakage, train-only preprocessing and held-out evaluation.

Assessment goal: Submit a short written or oral knowledge check plus evidence from your chapter work. Acceptance requires clear evidence of dataset shapes, a documented missing-value policy, preprocessing fitted on training data only, and a separate held-out dataset.

1. Assessment Overview

This assessment checks whether you can explain the core ideas from the NumPy, pandas and data-preparation chapter and demonstrate that you applied them correctly. The focus is not memorizing syntax. You should be able to explain what your preprocessing workflow does, why it avoids leakage, and how the shapes of your datasets change through the process.

ACCEPTANCE REQUIREMENTS

Shapes + Missing-value policy + Train-only fit + Held-out dataset

2. What You Must Submit

Required Evidence

A short written response or oral explanation answering the knowledge-check questions.
The raw feature and target shapes.
The training and held-out shapes after splitting.
Your documented policy for numeric, categorical and date-related missing values.
Code showing the preprocessor fitted only on training features.
Code showing the held-out data receives only transform().
The final processed training and held-out shapes.
Evidence that processed training and held-out feature columns are compatible.

3. Recommended Evidence Format

Use a compact evidence summary such as the following. Replace the example shapes with the actual values produced by your lab.

Evidence	Example	What it proves
Raw data	`(20, 7)`	The starting dataset dimensions are known.
Feature matrix	`X: (20, 6)`	The target has been separated from predictors.
Target	`y: (20,)`	One target value exists for each record.
Training features	`(15, 6)`	Training observations are isolated.
Held-out features	`(5, 6)`	A separate evaluation population exists.
Processed data	`(15, k)` and `(5, k)`	Both datasets use the same learned feature schema.

4. Knowledge Check Part A: Shapes and NumPy Reasoning

Question 1

A two-dimensional array has shape (100, 8). What do the two numbers mean?

Expected answer The array contains 100 rows or observations and 8 columns or features, assuming the dataset follows the common samples-by-features convention.

Question 2

If X.shape == (100, 8) and w.shape == (8,), what is the shape of X @ w?

SHAPE TRACE

\[ (100,8)@(8,) \rightarrow (100,) \]

Each of the 100 rows produces one dot-product result.

Question 3

If X.shape == (100, 8) and offset.shape == (8,), what happens with X + offset?

Expected answer NumPy broadcasts the eight-element vector across the 100 rows. The result keeps shape (100, 8).

Question 4

Does vectorizing an element-wise operation over \(n\) values normally change its asymptotic complexity from \(O(n)\) to \(O(1)\)?

Expected answer No. The operation still processes approximately \(n\) elements. Vectorization normally improves implementation efficiency by reducing Python-level loop overhead and using optimized numerical routines, while the element-wise operation remains \(O(n)\).

5. Knowledge Check Part B: pandas and Missing Values

Question 5

Why should you inspect df.dtypes after loading a CSV?

Expected answer CSV data types are inferred. A nominally numeric column can become text because of values such as "unknown", while dates may remain strings. Incorrect dtypes can cause invalid calculations or prevent intended preprocessing.

Question 6

Why should missing values not automatically be replaced with zero?

Expected answer Zero may be a legitimate observed value and may have a completely different meaning from missing. Missing-value treatment should depend on the feature's semantics and the reason values are absent.

Question 7

State a reasonable baseline missing-value policy for the lab.

Feature type	Policy	Fit scope
Numeric	Median imputation, optionally with a missing indicator	Training data only
Categorical	Most-frequent category or a documented explicit missing category	Training data only when a mode is learned
Date	Parse invalid values as missing, derive useful date components, then handle missing derived values through the numerical policy	Parsing is deterministic; statistical imputation is training only

6. Knowledge Check Part C: Leakage and Held-Out Data

Question 8

What is data leakage in preprocessing?

Expected answer Data leakage occurs when information unavailable at genuine prediction time influences training. For example, using held-out observations to calculate an imputation median or scaling mean allows evaluation data to influence the trained pipeline.

Question 9

Why must the train-test split occur before fitting an imputer or scaler?

CORRECT ORDER

Split → Fit preprocessing on train → Transform train → Transform held-out

The held-out set should simulate unseen data. Allowing it to influence preprocessing parameters weakens the independence of the final evaluation.

Question 10

What is the difference between fit_transform() and transform() in this workflow?

fit_transform()

Learns preprocessing parameters.
Transforms the same data.
Used on training features.

transform()

Does not relearn parameters.
Uses the already-fitted rules.
Used on held-out or future data.

Question 11

What is the role of the held-out dataset?

Expected answer It provides data that did not participate in model fitting or learned preprocessing. It is reserved for evaluating how the complete trained workflow behaves on unseen observations.

7. Evidence Check: Show Your Shapes

Include output similar to the following in your evidence submission.

print("Raw dataset:", clean_df.shape)
print("Feature matrix X:", X.shape)
print("Target y:", y.shape)
print("Training features:", X_train.shape)
print("Held-out features:", X_holdout.shape)
print("Training target:", y_train.shape)
print("Held-out target:", y_holdout.shape)
print("Processed training:", X_train_processed.shape)
print("Processed held-out:", X_holdout_processed.shape)

Acceptance evidence: The processed training and held-out datasets may have a larger column count because of one-hot encoding and missing indicators, but they must have the same final feature count and compatible column ordering.

8. Evidence Check: Show Your Missing-Value Policy

Your submission should explicitly describe what happens when values are missing. Do not submit only the final cleaned DataFrame.

EXAMPLE POLICY

Numeric → training median
Categorical → training mode
Unknown categories → ignore safely during one-hot transform
Invalid dates → coerce to missing, derive date features, impute statistically from training only

9. Evidence Check: Show Train-Only Fitting

The following two lines provide direct evidence of the required leakage boundary:

X_train_array = preprocessor.fit_transform(X_train_features)
X_holdout_array = preprocessor.transform(X_holdout_features)

Reject preprocessor.fit_transform(X_holdout_features)

Accept preprocessor.transform(X_holdout_features)

10. Evidence Check: Inspect Training-Fitted Values

For stronger evidence, show the statistics learned by the training-fitted preprocessing pipeline.

numeric_imputer = (
    preprocessor
    .named_transformers_["numeric"]
    .named_steps["imputer"]
)

categorical_imputer = (
    preprocessor
    .named_transformers_["categorical"]
    .named_steps["imputer"]
)

print("Numeric training statistics:")
print(numeric_imputer.statistics_)

print("Categorical training statistics:")
print(categorical_imputer.statistics_)

11. Evidence Check: Prove the Held-Out Dataset Exists

Show that held-out rows were created before preprocessing was fitted. You may also save the held-out snapshot as evidence.

holdout_evidence = X_holdout.copy()
holdout_evidence["purchased"] = y_holdout
holdout_evidence.to_csv("held_out_evidence.csv", index=False)

print("Held-out rows:", len(holdout_evidence))
print(holdout_evidence.head())

12. Evidence Check: Validate the Final Feature Schema

assert X_train_processed.shape[0] == len(y_train)
assert X_holdout_processed.shape[0] == len(y_holdout)

assert X_train_processed.shape[1] == X_holdout_processed.shape[1]
assert list(X_train_processed.columns) == list(X_holdout_processed.columns)

assert not X_train_processed.isna().any().any()
assert not X_holdout_processed.isna().any().any()

print("Evidence validation passed.")

13. Short Written Submission Template

You may use the following structure for your written submission and replace the placeholders with your actual outputs.

Dataset Shapes

My raw dataset shape was [rows, columns]. After separating the target, X had shape [...] and y had shape [...]. The training subset had shape [...] and the held-out subset had shape [...].

Missing-Value Policy

I converted known invalid tokens to missing values. Numerical missing values were imputed using medians learned from training data. Categorical missing values were handled using a documented training-fitted strategy. Invalid dates were parsed as missing before deterministic date-feature extraction.

Train-Only Preprocessing

I fitted the preprocessor using only the training features with fit_transform(). The held-out data was processed using only transform(), so it did not contribute to imputation, scaling or encoding parameters.

Held-Out Dataset

I created the held-out dataset before fitting learned preprocessing. It remained separate until the fitted training preprocessor was applied. The final training and held-out matrices had the same feature schema.

14. Short Oral Check

If completing the assessment orally, be prepared to answer the following in approximately one to two minutes each.

Oral Questions

Explain the difference between deterministic cleaning and learned preprocessing.
Explain why a held-out dataset must not determine an imputation median.
Describe the shapes of your raw, training and held-out datasets.
Describe your missing-value policy and why you selected it.
Explain why you used fit_transform() on training data and only transform() on held-out data.
Explain why one-hot encoding can change the number of columns.

15. Practical Code Check

Complete this short code task without fitting preprocessing on the held-out data.

# Assume these objects already exist:
# X_train_features
# X_holdout_features
# preprocessor

# TODO 1: fit preprocessing only on training data
X_train_processed = ______________________________

# TODO 2: apply the existing fitted preprocessing to held-out data
X_holdout_processed = ____________________________

# TODO 3: verify equal processed feature counts
assert ___________________________________________

Reference Answer

X_train_processed = preprocessor.fit_transform(X_train_features)
X_holdout_processed = preprocessor.transform(X_holdout_features)

assert X_train_processed.shape[1] == X_holdout_processed.shape[1]

16. NumPy and pandas Quick Check

Question	Expected concept
What does `array.shape` describe?	The size of each array axis.
What does broadcasting do?	Allows compatible differently shaped arrays to participate in element-wise operations.
What does `df.isna().sum()` show?	The number of missing values per column.
What is `.loc` primarily based on?	Labels and boolean conditions.
What is `.iloc` primarily based on?	Integer positions.
What does `merge()` do?	Combines related tabular datasets using key columns or indexes.
Why prefer vectorized arithmetic?	It expresses whole-array or whole-Series operations and usually reduces Python-level iteration overhead.

17. Assessment Acceptance Rubric

Criterion	Accepted when	Needs revision when
Shapes	Raw, train, held-out and processed shapes are clearly shown.	Shapes are omitted or inconsistent.
Missing-value policy	The treatment is documented by feature type and learned values come from training.	Missing values are filled without explaining the rule or fit scope.
Train-only preprocessing	The preprocessor is fitted using training data only.	The full or held-out dataset contributes to fit statistics.
Held-out dataset	It is created before statistical fitting and transformed afterward.	It is created after preprocessing or fitted separately.
Schema validation	Processed train and held-out feature counts or names are verified to match.	No compatibility check is shown.
Explanation	The learner can explain leakage, shapes and the preprocessing sequence.	The submission contains code without demonstrating conceptual understanding.

18. Self-Review Before Submission

Final Evidence Checklist

I included the raw dataset shape.
I separated features and target and showed both shapes.
I showed the training and held-out shapes.
I documented my missing-value policy.
I identified which preprocessing steps learn statistics.
I fitted those steps using training data only.
I used only transform() for held-out data.
I showed the final processed shapes.
I verified that processed training and held-out schemas match.
I included a short written or oral explanation of why this design prevents leakage.

Acceptance Standard

Your evidence is complete when you can show and explain dataset shapes, a defensible missing-value policy, train-only fitted preprocessing, and a genuinely held-out dataset. The key principle is that held-out information must not influence any preprocessing parameter learned during training.

Get Neural Networks and Deep Learning updates by email

Short, practical lessons and course updates from Rumman Ansari. No spam.