Introduction

Every system has a finite processing capacity. CPU, memory, threads, connections, queues, database locks, storage operations, and downstream service quotas all have limits.

When incoming work exceeds the rate at which the system can safely complete it, the system enters overload.

Incoming workload:

10,000 requests per second


Safe processing capacity:

6,000 requests per second


Excess workload:

4,000 requests per second

Without explicit overload control, excess requests commonly wait in queues, consume memory, occupy threads, hold connections, miss deadlines, and trigger client retries. Those retries further increase the incoming workload.

Traffic exceeds capacity
      |
      v
Queues grow
      |
      v
Latency increases
      |
      v
Requests time out
      |
      v
Clients retry
      |
      v
Traffic increases further
      |
      v
System throughput collapses

Rejecting a controlled number of requests early can be safer than accepting every request and allowing all of them to time out.

Prerequisites

What Is Overload?

Overload is a condition in which accepted work requires more of a constrained resource than the system can safely provide.

The constrained resource can be:

• CPU time
• Memory
• Application threads
• Event-loop capacity
• Database connections
• Database locks
• Storage operations
• Network bandwidth
• Queue capacity
• External API allowance
• One database partition or tenant-specific resource

A service can be overloaded even when the complete cluster has unused capacity. For example, one database partition, API route, tenant, or connection pool can be saturated while unrelated resources remain idle.

Overload vs Traffic Spike

Arrival Rate and Service Rate

Let \(\lambda\) represent the average arrival rate and \(\mu\) represent the safe completion rate.

A simplified stability condition is:

\[ \lambda < \mu \]

When work arrives faster than it can be completed for a sustained period:

\[ \lambda > \mu \]

Queue depth and waiting time will continue increasing unless the system adds capacity, slows producers, rejects work, or reduces the work required per request.

Queueing and Latency

A bounded queue can absorb a short burst. It cannot solve sustained capacity mismatch.

Short burst:

Queue temporarily grows
      |
      v
Arrival rate returns below capacity
      |
      v
Workers drain backlog


Sustained overload:

Queue continuously grows
      |
      v
Waiting time continuously increases

A simplified form of Little's Law is:

\[ WorkInSystem = Throughput \times AverageTimeInSystem \]

If queue depth increases while throughput remains limited, waiting time increases. Many queued requests can become useless because their client deadlines expire before processing begins.

Main Overload Controls

1. Admission control
2. Rate limiting
3. Concurrency limiting
4. Bounded queues
5. Backpressure
6. Load shedding
7. Request prioritization
8. Workload isolation
9. Circuit breaking
10. Deadlines and timeouts
11. Retry budgets
12. Graceful degradation
13. Caching and request coalescing
14. Autoscaling

Admission Control

Admission control decides whether new work should enter the system.

Incoming request
      |
      v
Is safe capacity available?
      |
      +-- Yes:
      |      accept request
      |
      +-- No:
             reject, defer,
             or redirect safely

The decision can consider:

• Active requests
• Queue depth
• Memory pressure
• Available database connections
• Tenant-specific capacity
• Dependency health
• Request priority
• Remaining request deadline

Admission control prevents work from consuming expensive resources when safe completion is unlikely.

Rate Limiting

Rate limiting controls how frequently a caller, tenant, route, or application can start requests.

Incoming request
      |
      v
Determine trusted limit key
      |
      v
Check rate-limiting policy
      |
      +-- Allowance available:
      |      continue
      |
      +-- Limit exceeded:
             reject or delay
             according to policy

Possible limit keys include:

• Authenticated account
• Tenant
• API subscription
• Client application
• API operation
• Trusted network identity
• A combination such as tenant and route

Rate-limit Response

HTTP/1.1 429 Too Many Requests
Content-Type: application/problem+json
Retry-After: 30

{
  "type": "about:blank",
  "title": "Request rate exceeded",
  "status": 429,
  "detail": "The caller exceeded the permitted request rate."
}

Return retry guidance only when the server can provide meaningful guidance. Clients should still use bounded retries and jitter.

Common Rate-limiting Algorithms

Token-bucket Model

A token bucket has a refill rate and a maximum token capacity.

Tokens refill at a controlled rate
      |
      v
Request arrives
      |
      +-- Token exists:
      |      consume token
      |      accept request
      |
      +-- Bucket empty:
             reject or delay request

The refill rate controls sustained traffic. Bucket capacity determines the permitted burst.

Fairness and Per-tenant Limits

One caller or tenant should not consume all shared capacity.

Shared API capacity
      |
      +-- Tenant A allowance
      +-- Tenant B allowance
      +-- Tenant C allowance
      +-- Protected shared reserve

Useful designs can combine:

• A fleet-wide safety limit
• A per-tenant limit
• A per-user limit
• A route-specific limit
• A protected reserve for critical operations

Use trusted identity context. A client-provided tenant or account identifier must not independently determine the applied limit.

Concurrency Limiting

Rate limits control how frequently work begins. Concurrency limits control how much work can remain active simultaneously.

Concurrency limit:

100 active report generations


Active operations:

100


New request:

Rejected, queued within a bound,
or deferred according to policy

Concurrency limiting is valuable when individual operations have variable or long duration.

Rate vs Concurrency Limit

Bounded Queues

A bounded queue defines the maximum amount of waiting work.

Producer
    |
    v
Bounded queue
    |
    v
Consumer


When queue is full:

- Block or slow producer
- Reject new work
- Drop work according to policy
- Redirect work to an approved path

The queue size should reflect useful waiting time, memory or storage capacity, workload value, and recovery objectives.

Backpressure

Backpressure communicates downstream capacity limitations to upstream producers.

Consumer slows down
      |
      v
Queue approaches its bound
      |
      v
Producer receives pressure signal
      |
      v
Producer slows, pauses,
or reduces request creation

Backpressure can appear as:

• A blocked bounded channel
• Reduced consumer demand
• A full queue signal
• A retryable overload response
• Transport flow control
• A reduced concurrency allowance

Load Shedding

Load shedding intentionally rejects, drops, abandons, or defers selected work when safe capacity is unavailable.

Active demand exceeds safe capacity
      |
      v
Classify incoming work
      |
      +-- Critical and feasible:
      |      process
      |
      +-- Optional:
      |      omit or degrade
      |
      +-- Expired:
      |      reject immediately
      |
      +-- Excess low-priority work:
             shed

Load shedding protects useful throughput by preventing excess work from consuming resources needed by requests that can still complete successfully.

Rate Limiting vs Backpressure vs Load Shedding

Request Prioritization

Not every request has the same business or operational importance.

Priority 1:

Authentication and authorization


Priority 2:

Course access and enrollment


Priority 3:

Search suggestions


Priority 4:

Analytics refresh and optional recommendations

During overload, the system can protect essential operations and reduce lower-priority work.

Priority must come from trusted server-side policy. A public client should not be able to label every request as critical.

Workload Isolation

Separate capacity pools can prevent one workload from exhausting resources needed by another.

Interactive API pool
    -> User-facing requests


Report-worker pool
    -> Expensive report generation


Media-worker pool
    -> Video and image processing


Administrative pool
    -> Controlled operations

Isolation can be applied by:

• Tenant
• API route
• Workload type
• Priority
• Region
• Queue
• Worker pool
• Database connection pool

Circuit Breakers

A circuit breaker stops repeated calls to a dependency that is failing or too slow.

Dependency calls succeed
      |
      v
Circuit closed


Failures exceed policy
      |
      v
Circuit opens
      |
      v
New calls fail fast
or use approved fallback


Recovery interval passes
      |
      v
Limited test calls
      |
      v
Close circuit after recovery

A circuit breaker protects caller resources such as threads, sockets, and request deadlines. It does not repair the failed dependency.

Deadlines and Timeouts

Requests should not consume resources after their useful deadline.

Client deadline:

2 seconds


Gateway work:

100 milliseconds


Service processing:

600 milliseconds


Database work:

500 milliseconds


Response transfer:

200 milliseconds


Remaining time:

Reserved for variation and failure handling

Each downstream operation should receive a timeout that fits within the remaining end-to-end deadline.

Deadline-aware Admission

A request already too old to complete successfully should not enter an expensive queue.

Request reaches queue boundary
      |
      v
Remaining deadline:

50 milliseconds


Expected queue and processing time:

500 milliseconds


Decision:

Reject or abandon before
consuming full processing capacity

Retry Budgets

Retries compete with new requests for the same capacity.

Original traffic:

5,000 requests per second


Every failed request retries twice

Potential attempts:

Up to 15,000 attempts per second

A retry budget limits retry traffic to a controlled portion of total attempts or available capacity.

Safe retry design includes:

• Bounded attempts
• Exponential backoff
• Randomized jitter
• End-to-end deadlines
• Idempotency
• Retry budgets
• Awareness of overload responses

Graceful Degradation

Graceful degradation preserves core functionality while reducing optional work.

Possible degraded modes include:

• Hide optional recommendations
• Return fewer search facets
• Serve approved stale public content
• Delay analytics updates
• Reduce image or report detail
• Queue nonessential notifications
• Disable expensive sorting or export options temporarily

The degraded response should be explicit where users or callers need to know that information is delayed, reduced, or temporarily unavailable.

Do Not Degrade Correctness Silently

Some operations require authoritative correctness.

Examples include:

• Payment processing
• Inventory reservation
• Authorization changes
• Financial posting
• Certificate issuance
• Enrollment confirmation requiring a committed transaction

When safe completion is unavailable, reject, defer, or queue the operation according to its contract instead of returning a false success.

Caching during Overload

Caching can reduce repeated work against application and database resources.

Popular request
      |
      v
Cache lookup
      |
      +-- Hit:
      |      return response
      |
      +-- Miss:
             request backend
             populate cache
             return result

Caching is useful only when the cached response is safe for the caller, tenant, authorization state, and required freshness.

Request Coalescing

Request coalescing allows concurrent requests for the same missing value to share one backend load.

1,000 simultaneous requests
for the same course
      |
      v
One backend retrieval
      |
      v
Result shared with
waiting requests

Without coalescing, a popular cache expiration can create a surge against the origin service.

Autoscaling and Overload Control

Autoscaling can add capacity, but it is not immediate.

Overload detected
      |
      v
Scaling decision
      |
      v
Provision instance
      |
      v
Start application
      |
      v
Pass readiness
      |
      v
Receive traffic

Rate limiting, admission control, bounded queues, and load shedding protect the service while additional capacity is starting.

Autoscaling also cannot fix:

• A hot database partition
• A serialized global lock
• An inefficient query
• A memory leak
• An exhausted external quota
• An unavailable dependency

HTTP Overload Responses

The selected response should accurately represent the failure contract. Avoid returning a successful status for work that was not accepted or completed.

Conceptual Overload Policy

overloadControl:
  admission:
    maximumConcurrentRequests: approved-limit
    maximumQueueDepth: approved-limit

  rateLimits:
    anonymous:
      policy: approved-anonymous-policy

    authenticated:
      key: trusted-caller
      policy: approved-caller-policy

    tenant:
      key: trusted-tenant
      policy: approved-tenant-policy

  priorities:
    critical:
      reservedCapacity: approved-reserve

    normal:
      admissionPolicy: standard

    optional:
      shedFirst: true

  retries:
    maximumAttempts: approved-limit
    exponentialBackoff: true
    jitter: true
    requireIdempotencyForWrites: true

  degradation:
    disableOptionalRecommendations: true
    deferAnalytics: true
    allowApprovedStalePublicContent: true

  observability:
    metrics: enabled
    structuredEvents: enabled
    tracing: enabled

This is conceptual configuration. Limits and policies must come from load tests, service objectives, dependency capacity, security requirements, and the selected platform.

PHP Concurrency Guard

<?php

declare(strict_types=1);

final class ConcurrencyGuard
{
    public function __construct(
        private ConcurrencyStore $store,
        private int $maximumConcurrentRequests
    ) {
    }

    public function execute(
        string $limitKey,
        callable $operation
    ): mixed {
        $acquired =
            $this->store->tryAcquire(
                $limitKey,
                $this->maximumConcurrentRequests
            );

        if (!$acquired) {
            throw new RuntimeException(
                'The service is currently at its safe concurrency limit.'
            );
        }

        try {
            return $operation();
        } finally {
            $this->store->release(
                $limitKey
            );
        }
    }
}

Production implementations require atomic acquisition, lease expiration, crash recovery, tenant isolation, metrics, and an approved failure policy.

PHP Overload Response

<?php

declare(strict_types=1);

function sendOverloadResponse(
    int $retryAfterSeconds
): void {
    http_response_code(
        503
    );

    header(
        'Content-Type: application/problem+json'
    );

    header(
        'Cache-Control: no-store'
    );

    header(
        'Retry-After: ' . $retryAfterSeconds
    );

    echo json_encode(
        [
            'type' => 'about:blank',
            'title' => 'Service temporarily unavailable',
            'status' => 503,
            'detail' =>
                'The service cannot safely accept additional work.'
        ],
        JSON_THROW_ON_ERROR
    );
}

A retry interval should be supplied only when the server has a reasonable basis for the value. Clients must still cap attempts and apply jitter.

Learning-platform Example

Incoming learning-platform traffic
      |
      v
API Gateway
      |
      +-- Per-caller rate limit
      +-- Per-tenant rate limit
      +-- Request-size limit
      |
      v
API Service
      |
      +-- Concurrency limit
      +-- Deadline validation
      +-- Optional feature shedding
      |
      v
Database and shared services

Suggested Priority Direction

Observability

Useful overload-control metrics include:

• Incoming request rate
• Accepted request rate
• Rejected request rate
• Requests by rejection reason
• Active request count
• Queue depth
• Oldest queued-work age
• Request latency by priority
• Timeout count
• Retry rate
• Circuit-breaker state
• Rate-limit usage by tenant and route
• Load-shedding count
• Degraded-response count
• Database connection usage
• Thread or worker-pool usage
• Cache hit rate
• Autoscaling capacity and delay

Structured Rejection Event

{
  "service": "course-api",
  "operation": "generate-report",
  "decision": "rejected",
  "reason": "concurrency-limit",
  "priority": "optional",
  "policyVersion": "approved-policy-version"
}

Do not record access tokens, session identifiers, private request bodies, or other reusable credentials in overload-control logs.

Alert Conditions

Alert when:

• Rejection rate increases unexpectedly
• Critical traffic is being shed
• Queue depth or age continues increasing
• Concurrency remains at its limit
• Retry traffic exceeds its budget
• Timeouts increase while useful throughput falls
• All instances enter overload protection
• A tenant repeatedly consumes its complete allowance
• A circuit breaker remains open beyond its expected recovery period
• Autoscaling reaches maximum capacity
• Load shedding does not restore latency or throughput
• Database or dependency limits are approached

Troubleshooting Workflow

1. Identify the saturated resource.
2. Compare arrival rate with successful completion rate.
3. Check active requests, queue depth, and oldest-work age.
4. Check timeouts and retry traffic.
5. Identify which tenant, route, key, or workload dominates demand.
6. Check rate and concurrency-limit decisions.
7. Check circuit-breaker states and dependency latency.
8. Check whether optional fan-out is still active.
9. Check load-balancer distribution and unhealthy instances.
10. Check autoscaling capacity, startup delay, and maximum limits.
11. Check database connections, locks, and hot partitions.
12. Confirm that rejected requests are not retried immediately.
13. Protect critical operations and reduce optional work.
14. Verify recovery gradually before restoring full traffic.

Common Overload-control Mistakes

Recommended Test Cases

Overload-control Best Practices

Practice Exercise

Design overload control for your online learning platform.

Requirements

1. Measure the safe request capacity of one API instance.
2. Set a fleet-wide concurrency boundary.
3. Create separate anonymous, authenticated, and tenant rate limits.
4. Reserve capacity for authentication and course access.
5. Classify recommendations and analytics as optional work.
6. Place report generation behind a bounded queue.
7. Limit concurrent report workers.
8. Propagate queue pressure to report producers.
9. Use deadlines for interactive requests.
10. Add circuit breakers for optional dependencies.
11. Add idempotency to enrollment submissions.
12. Apply bounded retries with backoff and jitter.
13. Add approved stale caching for public course information.
14. Add request coalescing for popular cache misses.
15. Test sustained overload after maximum autoscaling capacity is reached.
16. Verify that critical operations remain usable.
17. Verify gradual recovery after demand decreases.

Overload-control Design Template

Frequently Asked Questions