Prerequisites

What Rate Limiting Does

Rate limiting restricts how many operations a given caller may perform within a defined period. When the limit is exceeded, the request is rejected or delayed rather than served.

Here \(L\) is the permitted count and \(W\) is the window. The apparent simplicity is deceptive: the behaviour at window boundaries, across many servers, and under bursty traffic is where the real design work lies.

Why Systems Need It

Related but Distinct Controls

Fixed Window Counter

The simplest approach divides time into fixed intervals and counts requests within each. When the count exceeds the limit, further requests are rejected until the window resets.

A caller sending the full limit in the final moment of one window and again in the first moment of the next transmits twice the intended rate across a span shorter than a single window.

Sliding Window Log

This approach stores a timestamp for every request and counts those falling within the trailing window. It is exact, with no boundary artefact.

async function slidingWindowLog(key, limit, windowMs) {
    const now = Date.now();
    const windowStart = now - windowMs;
    const redisKey = `ratelimit:log:${key}`;

    const results = await redis
        .multi()
        .zremrangebyscore(redisKey, 0, windowStart)
        .zcard(redisKey)
        .zadd(redisKey, now, `${now}-${randomSuffix()}`)
        .pexpire(redisKey, windowMs)
        .exec();

    const countBeforeAdd = results[1];

    if (countBeforeAdd >= limit) {
        await redis.zremrangebyscore(redisKey, now, now);

        return {
            allowed: false,
            remaining: 0,
            retryAfterMs: windowMs
        };
    }

    return {
        allowed: true,
        remaining: limit - countBeforeAdd - 1,
        retryAfterMs: 0
    };
}

Sliding Window Counter

This hybrid approximates the sliding window by weighting the previous window's count according to how far the current window has progressed. It captures most of the accuracy at a fraction of the memory cost.

async function slidingWindowCounter(key, limit, windowMs) {
    const now = Date.now();
    const currentWindow = Math.floor(now / windowMs);
    const previousWindow = currentWindow - 1;
    const elapsedRatio = (now % windowMs) / windowMs;

    const currentKey = `ratelimit:${key}:${currentWindow}`;
    const previousKey = `ratelimit:${key}:${previousWindow}`;

    const [currentCount, previousCount] = await redis.mget(
        currentKey,
        previousKey
    );

    const estimated =
        Number(currentCount || 0) +
        Number(previousCount || 0) * (1 - elapsedRatio);

    if (estimated >= limit) {
        return {
            allowed: false,
            remaining: 0,
            resetMs: windowMs - (now % windowMs)
        };
    }

    await redis
        .multi()
        .incr(currentKey)
        .pexpire(currentKey, windowMs * 2)
        .exec();

    return {
        allowed: true,
        remaining: Math.floor(limit - estimated - 1),
        resetMs: windowMs - (now % windowMs)
    };
}

Token Bucket

A bucket holds tokens, refilled at a steady rate up to a maximum capacity. Each request consumes a token. An empty bucket means rejection.

Here \(B\) is bucket capacity, \(r\) is the refill rate, and \(\Delta t\) is elapsed time. Capacity determines burst tolerance while the refill rate determines sustained throughput.

const TOKEN_BUCKET_SCRIPT = `
local key        = KEYS[1]
local capacity   = tonumber(ARGV[1])
local refillRate = tonumber(ARGV[2])
local nowMs      = tonumber(ARGV[3])
local cost       = tonumber(ARGV[4])

local state    = redis.call('HMGET', key, 'tokens', 'updatedAt')
local tokens   = tonumber(state[1])
local updated  = tonumber(state[2])

if tokens == nil then
    tokens  = capacity
    updated = nowMs
end

local elapsed = math.max(0, nowMs - updated) / 1000
tokens = math.min(capacity, tokens + elapsed * refillRate)

local allowed = 0

if tokens >= cost then
    tokens  = tokens - cost
    allowed = 1
end

redis.call('HMSET', key, 'tokens', tokens, 'updatedAt', nowMs)
redis.call('PEXPIRE', key, math.ceil(capacity / refillRate * 1000) + 1000)

return { allowed, tokens }
`;

async function tokenBucket(key, config, cost = 1) {
    const [allowed, remaining] = await redis.eval(
        TOKEN_BUCKET_SCRIPT,
        1,
        `ratelimit:bucket:${key}`,
        config.capacity,
        config.refillRatePerSecond,
        Date.now(),
        cost
    );

    return {
        allowed: allowed === 1,
        remaining: Math.floor(remaining)
    };
}

Leaky Bucket

Requests enter a queue that drains at a constant rate. Overflow is rejected. The output rate is perfectly smooth, regardless of how irregular the input is.

Algorithm Selection

Choosing the Limiting Key

The identity used for counting determines whether the limit protects anything. A poorly chosen key either fails to stop abuse or punishes innocent users.

async function checkLayeredLimits(request) {
    const checks = [
        { key: `ip:${request.sourceIp}`,        limit: 600,  windowMs: 60_000 },
        { key: `user:${request.userId}`,        limit: 120,  windowMs: 60_000 },
        { key: `tenant:${request.tenantId}`,    limit: 5000, windowMs: 60_000 },
        { key: `endpoint:${request.userId}:${request.route}`, limit: 30, windowMs: 60_000 }
    ];

    for (const check of checks) {
        const result = await slidingWindowCounter(
            check.key,
            check.limit,
            check.windowMs
        );

        if (!result.allowed) {
            return {
                allowed: false,
                violatedScope: check.key.split(":")[0],
                resetMs: result.resetMs
            };
        }
    }

    return { allowed: true };
}

Cost-Based Limiting

Treating every request as equally expensive is a poor approximation. A bulk export and a status check consume vastly different resources, yet a simple counter charges them the same.

Distributed Enforcement

With many application instances, a purely local counter permits far more traffic than intended. The effective limit becomes the configured limit multiplied by the instance count.

Atomicity Is Mandatory

Failure Policy

Communicating Limits to Clients

A rejection that carries no information forces clients to guess, and guessing typically means retrying immediately, which worsens the overload.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
RateLimit-Limit: 120
RateLimit-Remaining: 0
RateLimit-Reset: 34
Retry-After: 34

{
    "error": "rate_limit_exceeded",
    "message": "Request limit exceeded for this endpoint.",
    "scope": "user",
    "limit": 120,
    "windowSeconds": 60,
    "retryAfterSeconds": 34,
    "requestId": "req-cc71e4"
}

Client-Side Backoff

Limits only work when clients respond sensibly. Synchronized retries produce a thundering herd at the moment the window resets.

async function callWithBackoff(operation, options) {
    let attempt = 0;

    while (attempt < options.maxAttempts) {
        const response = await operation();

        if (response.status !== 429) {
            return response;
        }

        attempt += 1;

        const serverHint = Number(
            response.headers.get("Retry-After")
        );

        const baseDelayMs = Number.isFinite(serverHint) && serverHint > 0
            ? serverHint * 1000
            : options.baseDelayMs * Math.pow(2, attempt - 1);

        const jitterMs = Math.random() * baseDelayMs * 0.3;

        await sleep(baseDelayMs + jitterMs);
    }

    throw new Error("Rate limit retries exhausted");
}

Abuse-Specific Limits

Some endpoints warrant far stricter treatment than general capacity limits, because the operation itself is the attack surface.

Threats and Mitigations

Monitoring

Common Design Mistakes

System Design Interview Discussion

Implementation Checklist

Knowledge Check

Summary

Rate limiting bounds how quickly a caller may consume a service, protecting capacity, enforcing fairness between tenants, and blunting abuse. It is distinct from quotas, throttling, and load shedding, which operate on different horizons.

Fixed windows are simple but permit double the limit at boundaries. Sliding window logs are exact but memory-hungry. Sliding window counters offer a practical balance, while token buckets permit controlled bursts and leaky buckets smooth output for fragile dependencies.

The limiting key matters as much as the algorithm. Address-only limits punish shared networks while missing distributed attacks, so layering address, user, and tenant dimensions is necessary. Cost weighting prevents callers from concentrating on the most expensive endpoints.

Across a fleet, counters must be shared and updated atomically, with a deliberate failure policy. Rejections should return 429 with limit, remaining, reset, and Retry-After, and clients must back off with jitter to avoid a synchronized retry storm.