1. Why NumPy and pandas Matter

Most data science and machine learning work begins long before model training. Real datasets contain incorrect data types, duplicate records, missing values, inconsistent labels and fields that need to be derived. NumPy provides efficient multidimensional arrays and numerical operations. pandas builds labeled tabular structures on top of that ecosystem and adds powerful tools for cleaning, grouping, joining and reshaping data.

A useful mental model is: NumPy is the numerical engine, while pandas is the labeled data toolbox. Understanding both makes data preparation faster, clearer and less error-prone.

2. Prerequisites and Setup

You should be comfortable with Python variables, lists, dictionaries, functions, comparisons and basic control flow. For the examples, use Python 3 with NumPy and pandas installed. A Jupyter Notebook, VS Code, PyCharm or another Python environment is suitable.

python -m pip install numpy pandas
python -c "import numpy, pandas; print(numpy.__version__, pandas.__version__)"

import numpy as np
import pandas as pd

print(np.__version__)
print(pd.__version__)

3. NumPy Arrays

A NumPy ndarray stores values in an N-dimensional, usually homogeneous array. Unlike a normal Python list, an array exposes shape, dimensionality and data type metadata and supports element-wise numerical operations.

import numpy as np

prices = np.array([120.0, 150.0, 90.0, 200.0])
matrix = np.array([[1, 2, 3], [4, 5, 6]])

print(prices.shape)   # (4,)
print(matrix.shape)   # (2, 3)
print(matrix.ndim)    # 2
print(matrix.size)    # 6
print(matrix.dtype)

3.1 Creating Arrays

Besides converting Python lists, NumPy can construct initialized arrays and numerical sequences. Using the correct constructor makes intent clear.

zeros = np.zeros((2, 3))
ones = np.ones((2, 3))
sequence = np.arange(0, 10, 2)
points = np.linspace(0, 1, 5)

print(sequence)  # [0 2 4 6 8]
print(points)    # [0.   0.25 0.5  0.75 1.  ]

3.2 Shape and Reshaping

Shape describes the length of every axis. Reshaping changes how the same values are organized. The total number of elements must remain compatible.

values = np.arange(12)
grid = values.reshape(3, 4)
print(grid)

flat = grid.ravel()
print(flat)

4. NumPy Indexing, Slicing and Boolean Selection

Indexing retrieves particular elements; slicing retrieves ranges. In two dimensions, use array[row, column]. Boolean masks are especially important because they let a condition select data without explicit loops.

a = np.array([10, 20, 30, 40, 50])
print(a[0])       # 10
print(a[-1])      # 50
print(a[1:4])     # [20 30 40]

m = np.array([[10, 11, 12],
              [20, 21, 22],
              [30, 31, 32]])
print(m[1, 2])    # 22
print(m[:, 1])    # second column
print(m[0:2, :])  # first two rows

mask = a >= 30
print(mask)
print(a[mask])    # [30 40 50]

source = np.array([1, 2, 3, 4])
view = source[1:3]
view[0] = 99
print(source)       # [ 1 99  3  4]

safe_copy = source[1:3].copy()

5. Broadcasting

Broadcasting allows NumPy to perform element-wise operations on arrays with compatible shapes without explicitly copying the smaller object. It is one of the central ideas behind concise numerical code.

prices = np.array([100.0, 200.0, 300.0])
with_tax = prices * 1.18
print(with_tax)  # [118. 236. 354.]

sales = np.array([[10, 20, 30],
                  [40, 50, 60]])
column_adjustment = np.array([1, 2, 3])
print(sales + column_adjustment)

In the second example, the shapes are (2, 3) and (3,). Starting from the right, both have length 3. NumPy conceptually applies the one-dimensional array across both rows.

5.1 An Incompatible Shape

a = np.ones((2, 3))
b = np.array([1, 2])
# a + b  # ValueError: shapes (2, 3) and (2,) are not broadcast-compatible

# Make the intended row-wise adjustment explicit:
b_column = b[:, np.newaxis]   # shape becomes (2, 1)
print(a + b_column)

6. Vectorization

Vectorization expresses operations over complete arrays or columns. It usually produces shorter code and delegates work to optimized implementations underneath NumPy and pandas.

prices = np.array([100, 200, 300, 400], dtype=float)

# Python-style loop
result = []
for price in prices:
    result.append(price * 1.18)

# Vectorized NumPy expression
result_vectorized = prices * 1.18
print(result_vectorized)

Vectorization is not simply “avoid every loop.” Some algorithms are inherently iterative. The practical rule is to prefer built-in array, Series, aggregation and string operations when they directly represent the task.

7. pandas Series and DataFrame

A Series is a one-dimensional labeled array. A DataFrame is a two-dimensional labeled table containing columns that may have different data types.

import pandas as pd

s = pd.Series([85, 92, 78], index=["Asha", "Ravi", "Mina"], name="score")
print(s)

df = pd.DataFrame({
    "customer_id": [101, 102, 103, 104],
    "city": ["Kolkata", "Delhi", "Kolkata", "Mumbai"],
    "sales": [1200.0, 900.0, 1500.0, 700.0]
})
print(df)

8. Inspect Data Before Cleaning It

Do not start changing values immediately. First inspect shape, column names, data types, missingness, duplicates and representative rows. This creates a baseline and helps prevent accidental transformations.

print(df.head())
print(df.shape)
print(df.columns)
print(df.dtypes)
print(df.info())
print(df.describe(include="all"))
print(df.isna().sum())
print(df.duplicated().sum())

9. pandas Selection and Indexing

Use column selection, .loc and .iloc deliberately. .loc is label based; .iloc is integer-position based.

df = pd.DataFrame({
    "customer_id": [101, 102, 103, 104],
    "city": ["Kolkata", "Delhi", "Kolkata", "Mumbai"],
    "sales": [1200, 900, 1500, 700]
})

print(df["sales"])
print(df[["customer_id", "sales"]])

# Label-based selection
print(df.loc[df["sales"] >= 1000, ["customer_id", "city", "sales"]])

# Position-based selection: first 3 rows, first 2 columns
print(df.iloc[:3, :2])

9.1 Combine Conditions Correctly

For pandas boolean masks, use & for AND, | for OR and ~ for NOT, with each comparison wrapped in parentheses.

high_kolkata = df[(df["city"] == "Kolkata") & (df["sales"] >= 1000)]
not_mumbai = df[~(df["city"] == "Mumbai")]
print(high_kolkata)

10. Data Types and Conversion

Correct data types are part of data quality. A price stored as text cannot reliably participate in numerical analysis, and a date stored as an arbitrary string makes chronological operations fragile.

raw = pd.DataFrame({
    "price": ["100.50", "bad", "250.00"],
    "order_date": ["2026-01-10", "2026-01-11", "invalid"]
})

raw["price"] = pd.to_numeric(raw["price"], errors="coerce")
raw["order_date"] = pd.to_datetime(raw["order_date"], errors="coerce")
print(raw)
print(raw.dtypes)

With errors="coerce", values that cannot be parsed become missing values. This is often safer in a cleaning pipeline because invalid entries become measurable instead of silently behaving like legitimate text.

11. Missing Values

Missing data may appear as NaN, None, NaT or pandas' nullable missing marker, depending on dtype. The first task is not filling missing values. It is understanding why they are missing and whether the absence carries information.

df = pd.DataFrame({
    "name": ["Asha", "Ravi", "Mina", "Kabir"],
    "age": [24, np.nan, 29, 35],
    "city": ["Kolkata", "Delhi", None, "Kolkata"],
    "income": [40000, 52000, np.nan, 61000]
})

print(df.isna())
print(df.isna().sum())
print(df.isna().mean().mul(100).round(1))

11.1 Dropping Missing Data

# Drop rows where age is missing
drop_age = df.dropna(subset=["age"])

# Keep rows having at least 3 non-missing fields
mostly_complete = df.dropna(thresh=3)

11.2 Filling Missing Data

filled = df.copy()
filled["age"] = filled["age"].fillna(filled["age"].median())
filled["city"] = filled["city"].fillna("Unknown")
filled["income_missing"] = filled["income"].isna()
filled["income"] = filled["income"].fillna(filled["income"].median())
print(filled)

12. Duplicates, Text Cleaning and Categories

Duplicate rows can inflate counts and distort aggregates. Inconsistent text such as "Kolkata", " kolkata " and "KOLKATA" can fragment a single category into several.

customers = pd.DataFrame({
    "customer_id": [1, 2, 2, 3],
    "city": [" Kolkata ", "DELHI", "DELHI", "mumbai"]
})

customers["city"] = customers["city"].str.strip().str.title()
customers = customers.drop_duplicates(subset=["customer_id"], keep="first")
print(customers)

13. Joins and Merges

Joins combine related tables using one or more keys. In pandas, merge() closely resembles SQL joins. Before merging, understand whether the key relationship is one-to-one, one-to-many or many-to-many.

orders = pd.DataFrame({
    "order_id": [1, 2, 3],
    "customer_id": [101, 102, 105],
    "amount": [500, 800, 300]
})

customers = pd.DataFrame({
    "customer_id": [101, 102, 103],
    "customer_name": ["Asha", "Ravi", "Mina"]
})

result = orders.merge(
    customers,
    on="customer_id",
    how="left",
    validate="many_to_one"
)
print(result)

13.1 Validate Join Cardinality

The validate argument is a valuable safety check. For example, many_to_one states that many rows may exist on the left for a key, but the right table should contain at most one matching row per key. If that assumption is violated, pandas raises an error instead of quietly multiplying rows.

enriched = orders.merge(
    customers,
    on="customer_id",
    how="left",
    validate="many_to_one",
    indicator=True
)
print(enriched["_merge"].value_counts())

13.2 Joining with Different Key Names

left = pd.DataFrame({"cust_id": [1, 2], "amount": [100, 200]})
right = pd.DataFrame({"customer_id": [1, 2], "segment": ["A", "B"]})

merged = left.merge(
    right,
    left_on="cust_id",
    right_on="customer_id",
    how="left"
)

14. Vectorized pandas Operations

pandas provides vectorized arithmetic, comparisons, string operations, datetime accessors, conditional selection and mapping. These should usually be your first choices for column transformations.

sales = pd.DataFrame({
    "quantity": [2, 5, 1, 4],
    "unit_price": [100.0, 60.0, 250.0, 80.0],
    "customer": ["  ASHA", "ravi ", "Mina", "kabir"]
})

sales["revenue"] = sales["quantity"] * sales["unit_price"]
sales["customer_clean"] = sales["customer"].str.strip().str.title()
sales["band"] = np.where(sales["revenue"] >= 300, "High", "Standard")
print(sales)

14.1 Avoid apply() When a Native Operation Exists

Series.apply() is useful for some custom logic, but it is not automatically vectorized. If arithmetic, .str, .dt, .map(), np.where(), np.select() or another built-in operation expresses the same transformation, prefer the native operation.

# Less direct for simple arithmetic
sales["revenue_apply"] = sales.apply(
    lambda row: row["quantity"] * row["unit_price"], axis=1
)

# Prefer the direct vectorized expression
sales["revenue"] = sales["quantity"] * sales["unit_price"]

15. Grouping and Aggregation

Data preparation often requires summarizing granular records. The split-apply-combine pattern groups rows, applies aggregations and returns a summarized result.

orders = pd.DataFrame({
    "city": ["Kolkata", "Delhi", "Kolkata", "Delhi"],
    "amount": [500, 800, 300, 400]
})

summary = (
    orders.groupby("city", as_index=False)
          .agg(
              total_sales=("amount", "sum"),
              average_sale=("amount", "mean"),
              order_count=("amount", "size")
          )
)
print(summary)

16. Dates and Times

Convert date columns early when date logic matters. Once a Series has a datetime dtype, the .dt accessor exposes year, month, day and other components.

orders = pd.DataFrame({
    "order_date": ["2026-01-05", "2026-02-12", "2026-02-20"]
})
orders["order_date"] = pd.to_datetime(orders["order_date"], errors="coerce")
orders["year"] = orders["order_date"].dt.year
orders["month"] = orders["order_date"].dt.month
orders["month_name"] = orders["order_date"].dt.month_name()
print(orders)

17. End-to-End Data Preparation Example

The following example combines inspection, cleaning, conversion, missing-value handling, joining, feature creation and validation. In a real project, preserve the raw input separately and write cleaned output to a new dataset.

import numpy as np
import pandas as pd

# Raw transactional data
orders = pd.DataFrame({
    "order_id": [1, 2, 3, 4, 4],
    "customer_id": [101, 102, 103, 105, 105],
    "order_date": ["2026-01-05", "2026-01-07", "bad-date", "2026-01-09", "2026-01-09"],
    "quantity": [2, 1, 3, 2, 2],
    "unit_price": [100.0, 250.0, np.nan, 80.0, 80.0]
})

customers = pd.DataFrame({
    "customer_id": [101, 102, 103, 104],
    "city": [" Kolkata ", "DELHI", "kolkata", "Mumbai"]
})

# 1. Remove duplicate business records
orders = orders.drop_duplicates(subset=["order_id"]).copy()

# 2. Parse types safely
orders["order_date"] = pd.to_datetime(orders["order_date"], errors="coerce")
orders["quantity"] = pd.to_numeric(orders["quantity"], errors="coerce")
orders["unit_price"] = pd.to_numeric(orders["unit_price"], errors="coerce")

# 3. Preserve missingness information, then impute for this demonstration
orders["unit_price_missing"] = orders["unit_price"].isna()
median_price = orders["unit_price"].median()
orders["unit_price"] = orders["unit_price"].fillna(median_price)

# 4. Standardize category text
customers["city"] = customers["city"].str.strip().str.title()

# 5. Enrich orders with customer data and validate key cardinality
prepared = orders.merge(
    customers,
    on="customer_id",
    how="left",
    validate="many_to_one",
    indicator=True
)

# 6. Build vectorized features
prepared["revenue"] = prepared["quantity"] * prepared["unit_price"]
prepared["large_order"] = prepared["revenue"] >= 300
prepared["order_month"] = prepared["order_date"].dt.to_period("M").astype("string")

# 7. Validate the output
assert prepared["order_id"].is_unique
assert (prepared["quantity"].dropna() >= 0).all()

print(prepared)
print(prepared.isna().sum())
print(prepared["_merge"].value_counts())

18. Data Preparation Validation

A pipeline is not complete just because it runs. Add checks for assumptions that must remain true. Assertions are useful during development, while production systems may use dedicated validation frameworks or explicit error handling.

# Examples of simple quality checks
assert df.columns.is_unique
assert df["customer_id"].notna().all()
assert (df["sales"] >= 0).all()

# Check uniqueness without stopping execution immediately
if not df["customer_id"].is_unique:
    print("Warning: duplicate customer IDs detected")

19. Performance and Memory Practices

For moderate and large datasets, both algorithm choices and dtypes matter. Avoid unnecessary copies, select only needed columns when loading files, use vectorized expressions, and profile before optimizing. Converting low-cardinality repeated strings to categorical data can reduce memory in suitable cases.

# Inspect memory consumption
print(df.memory_usage(deep=True))

# Example category conversion when values repeat heavily
df["city"] = df["city"].astype("category")

# Prefer explicit columns instead of keeping unnecessary data
small = df[["customer_id", "city", "sales"]].copy()

20. NumPy and pandas: When to Use Which?

21. Mathematical View of Standardization

A common numerical preprocessing step is standardization. For a value \(x\), mean \(\mu\), and standard deviation \(\sigma\), the standardized score is:

This transformation centers a feature around zero and expresses values in units of standard deviation. For machine learning, estimate \(\mu\) and \(\sigma\) using training data only.

x = np.array([10.0, 20.0, 30.0, 40.0])
mean = x.mean()
std = x.std()
z = (x - mean) / std
print(z)

22. Practice Exercises

23. Practical Data Preparation Workflow