1. Prerequisites

This lesson assumes that you already understand basic Python syntax and have introductory knowledge of NumPy arrays. You should be comfortable reading variables, loops, lists, mathematical expressions and simple Python functions.

import numpy as np

print(np.__version__)

2. Why Shape Reasoning Matters

Many numerical programming errors are not caused by complicated mathematics. They are caused by incompatible shapes. A formula may be mathematically correct but still fail because the programmer represented the data with the wrong dimensions.

When working with NumPy, pandas, machine learning or scientific computing, develop the habit of asking three questions before performing an operation:

3. Understanding Array Shape

The shape of an array is a tuple describing the number of elements along each axis.

import numpy as np

scalar = np.array(10)
vector = np.array([10, 20, 30])
matrix = np.array([
    [1, 2, 3],
    [4, 5, 6]
])

print(scalar.shape)   # ()
print(vector.shape)   # (3,)
print(matrix.shape)   # (2, 3)

4. A Vector Is Not Automatically a Column Vector

This distinction is extremely important. A one-dimensional NumPy array with shape (3,) is neither a two-dimensional row matrix nor a two-dimensional column matrix.

x = np.array([10, 20, 30])

row = x.reshape(1, 3)
column = x.reshape(3, 1)

print(x.shape)       # (3,)
print(row.shape)     # (1, 3)
print(column.shape)  # (3, 1)

5. Trace Shapes Through a Computation

Shape tracing means writing the shape beside each variable as you follow an expression. This is one of the simplest ways to debug numerical code.

Consider a dataset with 4 observations and 3 features:

X = np.array([
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9],
    [10, 11, 12]
])

weights = np.array([0.2, 0.5, 0.3])

print(X.shape)        # (4, 3)
print(weights.shape)  # (3,)

scores = X @ weights
print(scores.shape)   # (4,)

5.1 Shape Derivation

For the example:

The three feature values in each row are combined with the three weights, producing one score per observation. Four input rows therefore produce four output values.

6. General Matrix Multiplication Shape Rule

Suppose matrix \(A\) has shape \((m,n)\) and matrix \(B\) has shape \((n,p)\). Matrix multiplication is valid because the inner dimensions are equal.

The inner dimension \(n\) participates in the summation. The outer dimensions \(m\) and \(p\) determine the result shape.

A = np.ones((5, 3))
B = np.ones((3, 2))
C = A @ B

print(A.shape)  # (5, 3)
print(B.shape)  # (3, 2)
print(C.shape)  # (5, 2)

6.1 Why the Inner Dimensions Must Match

Each output element is a dot product between one row of \(A\) and one column of \(B\):

Both operands must therefore contain the same number \(n\) of values along the contracted dimension.

7. Element-Wise Operations Versus Matrix Multiplication

Do not confuse * with @. Their mathematical meanings and shape requirements are different.

A = np.array([
    [1, 2],
    [3, 4]
])

B = np.array([
    [5, 6],
    [7, 8]
])

print(A * B)  # element-wise multiplication
print(A @ B)  # matrix multiplication

8. Broadcasting Shape Reasoning

Broadcasting allows element-wise operations between arrays with different but compatible shapes. NumPy compares dimensions from right to left.

8.1 Example: Add a Feature Offset

X = np.array([
    [10, 20, 30],
    [40, 50, 60],
    [70, 80, 90],
    [100, 110, 120]
])

offset = np.array([1, 2, 3])

print(X.shape)       # (4, 3)
print(offset.shape)  # (3,)

result = X + offset
print(result.shape)  # (4, 3)

Trace the dimensions from the right:

The final dimension is 3 in both operands. The missing leading dimension behaves as though the smaller array were available for every row.

8.2 Example: Column-Wise Broadcasting

X = np.ones((4, 3))
row_adjustment = np.array([10, 20, 30, 40]).reshape(4, 1)

print(X.shape)               # (4, 3)
print(row_adjustment.shape)  # (4, 1)

result = X + row_adjustment
print(result.shape)          # (4, 3)

The second dimension expands conceptually from 1 to 3.

8.3 Broadcasting Failure

A = np.ones((4, 3))
B = np.ones((4,))

# A + B
# ValueError because the trailing dimensions 3 and 4 are incompatible.

B = np.ones((4, 1))
result = A + B
print(result.shape)  # (4, 3)

9. Shape Tracing for Data Preprocessing

Shape reasoning becomes especially useful when preprocessing features for machine learning.

Assume:

• \(m\) = number of observations
• \(n\) = number of features
• \(X\) = feature matrix with shape \((m,n)\)
• \(\mu\) = feature means with shape \((n,)\)
• \(\sigma\) = feature standard deviations with shape \((n,)\)

Standardization is:

Shape trace:

X = np.array([
    [10.0, 100.0],
    [20.0, 150.0],
    [30.0, 200.0]
])

mu = X.mean(axis=0)
sigma = X.std(axis=0)
Z = (X - mu) / sigma

print(X.shape)      # (3, 2)
print(mu.shape)     # (2,)
print(sigma.shape)  # (2,)
print(Z.shape)      # (3, 2)

10. Loop-Based Computation

Consider scaling every element of an array containing \(n\) values.

values = [1, 2, 3, 4, 5]
result = []

for value in values:
    result.append(value * 2)

The loop performs one multiplication for every input element.

The constants \(a\) and \(b\) represent implementation costs such as executing Python bytecode, retrieving elements, multiplying values and appending results. Big-O notation ignores those constants when describing asymptotic growth.

11. Vectorized Computation

The equivalent NumPy operation can be expressed as one array statement:

values = np.array([1, 2, 3, 4, 5])
result = values * 2

This operation still has to process \(n\) values. Therefore its asymptotic computational complexity is also:

12. Same Big-O Does Not Mean Same Runtime

A Python loop and a vectorized NumPy operation may both be \(O(n)\), but their constants can differ significantly.

For suitable numerical operations, \(c_v\) is often much smaller than \(c_p\). NumPy performs repeated homogeneous numerical work in optimized compiled routines rather than executing the body of a Python loop for every element.

13. Complexity of Common Array Operations

14. Deriving Matrix-Vector Complexity

Let \(X\) have shape \((m,n)\) and \(w\) have shape \((n,)\).

There are \(m\) output values. Computing each output requires processing approximately \(n\) feature-weight pairs.

15. Deriving Matrix-Matrix Complexity

For conventional matrix multiplication:

The output contains \(m \times p\) elements. Each output element requires a dot product across \(n\) values.

For two square \(n \times n\) matrices, this conventional algorithm becomes:

This \(O(n^3)\) expression describes the conventional algorithm. Highly optimized numerical libraries may choose different algorithms or hardware-specific implementations for particular matrix sizes.

16. Loop Implementation Versus Matrix Operation

16.1 Explicit Python Loops

X = np.array([
    [1.0, 2.0, 3.0],
    [4.0, 5.0, 6.0]
])
w = np.array([0.2, 0.5, 0.3])

scores = []

for i in range(X.shape[0]):
    total = 0.0
    for j in range(X.shape[1]):
        total += X[i, j] * w[j]
    scores.append(total)

print(scores)

The outer loop runs \(m\) times and the inner loop runs \(n\) times.

16.2 Vectorized Version

scores = X @ w
print(scores)

The mathematical work remains proportional to \(mn\):

The vectorized statement is shorter and allows NumPy to delegate the numerical operation to optimized lower-level routines.

17. Benchmarking Loop and Vectorized Code

Use measurement when performance matters. The following example gives a simple benchmark. Actual timings depend on CPU, NumPy build, array size, dtype, memory layout and system load.

import time
import numpy as np

n = 1_000_000
values = np.arange(n, dtype=np.float64)

start = time.perf_counter()
loop_result = []
for value in values:
    loop_result.append(value * 2.0)
loop_time = time.perf_counter() - start

start = time.perf_counter()
vector_result = values * 2.0
vector_time = time.perf_counter() - start

print(f"Python loop: {loop_time:.6f} seconds")
print(f"NumPy vectorized: {vector_time:.6f} seconds")
print(f"Speed ratio: {loop_time / vector_time:.2f}x")

18. Why Vectorization Can Be Faster

19. Vectorization Has a Memory Cost

Vectorization is powerful, but an expression may allocate temporary arrays. For large datasets, memory complexity can become as important as time complexity.

x = np.arange(10_000_000, dtype=np.float64)

# This expression may require intermediate array storage.
y = (x * 2.0) + 10.0

An array of \(n\) 64-bit floating-point values requires approximately:

For \(n=10,000,000\):

That is about 80 MB using decimal units, or approximately 76.3 MiB using binary units, for one such raw array.

20. Broadcasting Can Avoid Explicit Replication

One advantage of broadcasting is that you normally do not need to manually duplicate the smaller operand just to match the larger shape.

X = np.ones((1000, 3))
offset = np.array([10, 20, 30])

# Normally unnecessary:
repeated = np.tile(offset, (1000, 1))
result = X + repeated

result = X + offset

21. Looping Is Not Always Wrong

Vectorization should not become a rule that makes code less correct or less understandable. Python loops remain appropriate when computation contains irregular control flow, stateful dependencies or operations that do not have a suitable vectorized equivalent.

22. pandas: Loop Versus Vectorized Column Computation

The same principle applies to pandas. If a calculation can be expressed with whole Series, prefer the Series expression over row iteration.

import pandas as pd

sales = pd.DataFrame({
    "quantity": [2, 5, 3, 4],
    "unit_price": [100.0, 50.0, 80.0, 120.0]
})

# Vectorized column expression
sales["revenue"] = sales["quantity"] * sales["unit_price"]

print(sales)

For \(n\) rows, the multiplication still requires work proportional to the number of rows:

23. A Complete Shape-Tracing Example

Suppose a small neural-style linear transformation contains:

• 32 observations
• 10 input features
• 4 output features

Represent the arrays as:

The transformation is:

23.1 Step 1: Matrix Multiplication

23.2 Step 2: Add the Bias

23.3 Complexity

The dominant matrix multiplication performs work proportional to:

More generally, for batch size \(m\), input dimension \(n\) and output dimension \(p\):

X = np.random.default_rng(42).normal(size=(32, 10))
W = np.random.default_rng(43).normal(size=(10, 4))
b = np.zeros(4)

Y = X @ W + b

print(X.shape)  # (32, 10)
print(W.shape)  # (10, 4)
print(b.shape)  # (4,)
print(Y.shape)  # (32, 4)

24. Design Assumptions You Should State

Complexity and shape analysis become much clearer when assumptions are explicit.

25. Common Shape Mistakes

26. A Repeatable Shape-Tracing Method

27. Worked Exercise

Given:

Find the shape and conventional computational complexity of:

Solution

First multiply \(X\) and \(W\):

Then broadcast \(b\) across the 1000 rows:

The conventional matrix multiplication dominates the computation:

The bias addition itself processes 8000 output values, so it is \(O(1000 \times 8)\). The matrix multiplication is the larger term in this design.

28. Practice Questions

29. Final Checklist