Introduction

Latency and throughput are two fundamental performance measurements in system design. They are related, but they answer different questions.

• Latency asks how long one operation takes.
• Throughput asks how many operations are completed during a period.

A system can have high throughput while individual requests remain slow. A system can also process one request very quickly while supporting only a small number of requests per second.

In the System Design curriculum, latency versus throughput is Topic 1.6 under System Design Process and Estimation. It follows high-level diagrams and precedes availability, reliability, QPS, storage and bandwidth estimation.

Prerequisites

What Is Latency?

Latency is the elapsed time between the beginning of an operation and the observation of its result at a defined measurement boundary.

Examples include:

• Time from submitting a search to receiving results
• Time from sending an API request to receiving the response
• Time required to execute a database query
• Time from publishing a message until a consumer receives it
• Time from submitting a notification until delivery

Latency Units

Components of End-to-End Latency

End-to-end latency is normally composed of several delays across the request path.

\[ L_{total} = L_{client} + L_{network} + L_{queue} + L_{application} + L_{dependencies} + L_{response} \]

Sequential Dependencies

When dependencies are called sequentially, their delays contribute one after another to the critical path.

\[ L_{sequential} = L_1 + L_2 + L_3 + \cdots + L_n \]

Client
  |
  | 20 ms
  v
Application
  |
  | 30 ms
  v
Database
  |
  | 40 ms
  v
External Service

Simplified total = 20 + 30 + 40 = 90 ms

Parallel Dependencies

Independent calls can sometimes run concurrently. The dependency portion of latency is then influenced by the slowest required branch rather than by adding every branch duration.

\[ L_{parallel} \approx \max(L_1, L_2, \ldots, L_n) + L_{coordination} \]

             +--> Service A: 30 ms --+
Application -+--> Service B: 50 ms --+--> Combine results
             +--> Service C: 40 ms --+

Dependency path is influenced by the 50 ms branch,
plus coordination and response assembly.

Parallel calls can reduce elapsed time, but they can increase downstream load and introduce partial-failure handling.

What Is Throughput?

Throughput is the amount of successfully completed work produced by a system during a defined period.

\[ Throughput = \frac{Completed\ Operations}{Time} \]

Throughput can be expressed as:

• Requests per second
• Transactions per second
• Messages processed per second
• Jobs completed per minute
• Records processed per hour
• Bytes transferred per second

Latency vs Throughput

Highway Analogy

A highway provides a useful conceptual analogy:

• Latency is how long one vehicle takes to complete the journey.
• Throughput is how many vehicles complete the journey per hour.
• Capacity or bandwidth is the maximum potential volume supported by the road.

A wide road can have high potential capacity but poor actual throughput during congestion. One vehicle can also complete a journey quickly while the road carries very few vehicles overall.

Average Latency Is Not Enough

Request latency is a distribution. Different requests can take different amounts of time due to queues, cache misses, storage delays, retries, garbage collection, network variation or resource contention.

Latency Percentiles

Percentiles reveal slow requests that an average can hide. The measurement report should also define the sampling period, workload and observation boundary.

Average Can Hide Tail Latency

Nine requests complete in 10 ms.
One request completes in 910 ms.

Average latency:
(9 x 10 + 910) / 10 = 100 ms

The average does not show that most requests were fast
while one request was much slower.

Concurrency

Concurrency is the amount of work in progress at the same time. Concurrency can increase throughput by allowing independent operations to make progress simultaneously.

However, increasing concurrency beyond resource limits can increase queueing, contention, timeouts and failures.

Little’s Law

For a stable system, Little’s Law relates the average number of operations in the system, average completion rate and average time spent in the system.

\[ L = \lambda W \]

Where:

• \(L\) is the average work in progress
• \(\lambda\) is the average throughput
• \(W\) is the average time in the system

Example

If a system completes 200 requests per second and average end-to-end latency is 0.25 seconds:

\[ L = 200 \times 0.25 = 50 \]

The system has approximately 50 requests in progress on average under those stated conditions.

Queueing and Saturation

A request can spend significant time waiting before processing begins. When work arrives faster than the limiting stage can complete it, a queue grows.

\[ Queue\ Growth\ Rate = Arrival\ Rate - Completion\ Rate \]

Arrival rate:    1,200 requests/second
Completion rate: 1,000 requests/second

Queue growth:
1,200 - 1,000 = 200 requests/second

If the difference continues, waiting time, timeouts and memory consumption can increase until requests are rejected or the system fails.

Typical Saturation Pattern

Low load:
Low queueing
Stable latency
Available spare capacity

Rising load:
Higher throughput
Increasing resource utilization

Near saturation:
Queueing rises
Tail latency increases
Little spare capacity

Overload:
Arrival exceeds sustainable completion
Timeouts and failures increase
Throughput can stop improving or decline

The Bottleneck Determines Throughput

A multi-stage system can process work only as fast as its limiting stage over a sustained period.

Gateway capacity:       10,000 requests/second
Application capacity:    5,000 requests/second
Database capacity:       2,000 requests/second
External provider:       1,000 requests/second

The external provider is the limiting stage for a flow
that requires the provider for every completed request.

Adding capacity to a non-limiting stage does not necessarily improve end-to-end throughput.

Can Throughput Increase Without Lower Latency?

Yes. Batching and concurrency can increase completed work without reducing the time required for an individual request. They can sometimes increase individual latency.

Batching Example

Without batching:
Process one event per storage operation.

With batching:
Collect several events.
Write the batch in one storage operation.

Possible effect:
Higher throughput
Fewer storage operations
Additional waiting while the batch fills

Batching is appropriate when throughput gains justify the additional delay and the workload permits grouped processing.

Strategies to Reduce Latency

Strategies to Increase Throughput

Example: URL Shortener

A URL shortener contains at least two performance-sensitive flows with different characteristics.

Create-link Flow

Client
  |
  v
Gateway
  |
  v
Link Service
  |
  +--> Validate destination
  +--> Generate short code
  +--> Write mapping
  |
  v
Link Store
  |
  v
Response

Important measurements include:

• End-to-end creation latency
• Creation requests completed per second
• Storage-write latency
• Duplicate or conflict rate
• Error rate under peak load

Redirect Flow

Visitor
  |
  v
Edge / Gateway
  |
  v
Redirect Service
  |
  +--> Cache hit ----------> Redirect response
  |
  +--> Cache miss
           |
           v
       Link Store
           |
           v
       Populate cache
           |
           v
       Redirect response

Important measurements include:

• End-to-end redirect latency
• Redirects completed per second
• Cache-hit and cache-miss latency
• Cache-hit ratio
• Link-store query rate
• Tail latency under peak load
• Timeout and error rate

Performance Interpretation

A high cache-hit ratio may reduce average database workload and improve redirect latency. However, the design must still define cache-miss behaviour, staleness, invalidation and cache-failure handling.

Analytics processing can be removed from the redirect critical path when synchronous analytics completion is not required. This can improve user-facing latency, but introduces asynchronous-delivery, backlog and duplicate-processing considerations.

Writing Performance Requirements

The URL shortener must be extremely fast and handle many users.

Under:
<defined workload, data set and environment>

The operation:
<specific request or background job>

Shall achieve:
<latency percentile, throughput and error-rate targets>

Measured from:
<defined entry and completion boundaries>

For:
<defined test duration>

Numeric targets should come from stakeholder requirements, existing measurements, approved forecasts or documented assumptions.

Performance Testing

Test Configuration Template

Test name:
<Descriptive name>

Operation:
<Endpoint, query, message or job>

Environment:
<Infrastructure and configuration>

Data set:
<Size and distribution>

Workload:
<Arrival rate, concurrency and request mix>

Duration:
<Warm-up and measurement periods>

Measurements:
- Successful throughput
- p50 latency
- p95 latency
- p99 latency
- Error rate
- Timeout rate
- Queue depth
- Resource utilization

Pass criteria:
<Approved measurable targets>

Monitor Latency and Throughput Together

Throughput should be interpreted together with latency, failures, queue depth and resource saturation.

Performance Review Checklist

Common Mistakes

Practice Exercise

Analyze the latency and throughput of a notification service that accepts requests through an API and delivers them asynchronously.

Flow

Client
  |
  v
Notification API
  |
  v
Notification Store
  |
  v
Delivery Queue
  |
  v
Channel Worker
  |
  v
External Provider
  |
  v
Delivery Status Store

Tasks

1. Define API acceptance latency.
2. Define end-to-end delivery latency.
3. Define accepted-request throughput.
4. Define successful-delivery throughput.
5. Identify the likely limiting stages.
6. Determine which queue measurements are required.
7. Explain how batching could affect throughput and latency.
8. Explain how provider rate limits affect sustainable throughput.
9. Create a load-test configuration.
10. Define the measurements required to identify saturation.

Model Analysis

Frequently Asked Questions