Prerequisites

The Defining Difference

In a single process, a failure is total. The program either completes an operation or crashes, and if it crashes, nothing downstream of that point executed. This binary property makes local reasoning tractable.

A distributed system has no such property. Some components succeed while others fail, and crucially, the surviving components cannot reliably determine which is which.

The Uncertainty of a Timeout

When a request times out, the caller knows only that no response arrived within the allotted time. Every one of the following remains possible.

The Two Generals Problem

Two generals must attack simultaneously to win. They communicate only by messenger across enemy territory, where any messenger may be captured. Can they reach certain agreement?

The first general sends a plan but cannot know it arrived. An acknowledgement helps, but the second general cannot know the acknowledgement arrived. Each confirmation requires its own confirmation, without end.

Failure Detection Is Always a Guess

Since a slow node is indistinguishable from a dead one, every failure detector must choose a threshold. That choice trades two types of error against each other.

Health Checks Lie

A health check that returns success proves only that the health endpoint responded. It says nothing about whether the service can perform real work.

Gray Failure

The most troublesome failures are not clean crashes. A component that is intermittently slow, partially correct, or failing only for certain requests evades every binary detector.

Network Partitions

A partition splits the system so that groups of nodes can communicate internally but not across the divide. Each side observes the other as failed.

Cascading Failure

Partial failure becomes total failure when the system's response to trouble generates more trouble. The most common amplifier is retrying.

With three layers each retrying three times, one client request can produce twenty-seven requests at the innermost service. A dependency that is merely slow receives an order of magnitude more load precisely when it can least handle it.

Bulkheads

Named after ship compartments, bulkheads isolate resource pools so exhaustion in one area cannot consume capacity needed elsewhere.

class Bulkhead {
    constructor(name, maxConcurrent, maxQueued) {
        this.name = name;
        this.maxConcurrent = maxConcurrent;
        this.maxQueued = maxQueued;
        this.active = 0;
        this.queued = 0;
    }

    async execute(operation) {
        if (this.active >= this.maxConcurrent) {
            if (this.queued >= this.maxQueued) {
                throw new BulkheadRejection(
                    `${this.name} at capacity`
                );
            }

            this.queued += 1;
            await this.waitForSlot();
            this.queued -= 1;
        }

        this.active += 1;

        try {
            return await operation();
        } finally {
            this.active -= 1;
            this.releaseSlot();
        }
    }
}

const bulkheads = {
    payments:        new Bulkhead("payments", 40, 20),
    recommendations: new Bulkhead("recommendations", 10, 0),
    search:          new Bulkhead("search", 30, 15)
};

Without isolation, a slow recommendation service can occupy every available thread, taking down checkout alongside it. With isolation, recommendations fail while payments continue.

Circuit Breakers

A circuit breaker stops calling a failing dependency, failing fast instead of accumulating waiting requests. It converts slow failure into immediate failure, which is far less damaging.

class CircuitBreaker {
    constructor(config) {
        this.state = "closed";
        this.failureCount = 0;
        this.successCount = 0;
        this.openedAt = 0;
        this.config = config;
    }

    async call(operation, fallback) {
        if (this.state === "open") {
            const elapsed = Date.now() - this.openedAt;

            if (elapsed < this.config.cooldownMs) {
                return fallback
                    ? await fallback()
                    : Promise.reject(new CircuitOpenError());
            }

            this.state = "half-open";
            this.successCount = 0;
        }

        try {
            const result = await operation();
            this.recordSuccess();
            return result;
        } catch (error) {
            this.recordFailure();

            if (fallback) {
                return await fallback();
            }

            throw error;
        }
    }

    recordSuccess() {
        if (this.state === "half-open") {
            this.successCount += 1;

            if (this.successCount >= this.config.probesToClose) {
                this.state = "closed";
                this.failureCount = 0;
            }
            return;
        }

        this.failureCount = 0;
    }

    recordFailure() {
        this.failureCount += 1;

        if (this.state === "half-open") {
            this.state = "open";
            this.openedAt = Date.now();
            return;
        }

        if (this.failureCount >= this.config.failureThreshold) {
            this.state = "open";
            this.openedAt = Date.now();
        }
    }
}

Timeout Budgets

Timeouts configured independently at each layer produce incoherent behaviour. An inner timeout longer than the outer one means the caller has already abandoned the request while the inner service continues consuming resources.

async function callWithDeadline(request, operation) {
    const remainingMs = request.deadline - Date.now();

    if (remainingMs <= MINIMUM_USEFUL_MS) {
        throw new DeadlineExceededError(
            "Insufficient time remaining to attempt call"
        );
    }

    const budgetMs = Math.min(
        remainingMs - RESERVE_FOR_RESPONSE_MS,
        MAX_SINGLE_CALL_MS
    );

    return await operation({
        timeoutMs: budgetMs,
        deadline: request.deadline
    });
}

Idempotency as the Foundation

Because a caller can never know whether an operation completed, safe retry requires that repeating an operation produces the same outcome as performing it once.

CREATE TABLE idempotency_records (
    idempotency_key VARCHAR(80)  PRIMARY KEY,
    operation_type  VARCHAR(60)  NOT NULL,
    request_hash    VARCHAR(64)  NOT NULL,
    status          VARCHAR(20)  NOT NULL,
    response_body   JSONB        NULL,
    created_at      TIMESTAMP    NOT NULL,
    completed_at    TIMESTAMP    NULL,
    expires_at      TIMESTAMP    NOT NULL
);

-- Claim the key atomically; a conflict means a retry
INSERT INTO idempotency_records (
    idempotency_key, operation_type, request_hash,
    status, created_at, expires_at
)
VALUES (
    :key, :operation, :hash,
    'in_progress', CURRENT_TIMESTAMP,
    CURRENT_TIMESTAMP + INTERVAL '24 hours'
)
ON CONFLICT (idempotency_key) DO NOTHING;

Graceful Degradation

Not every dependency is essential to every request. Designing which features may be sacrificed converts partial failure into reduced functionality rather than an error page.

Failure Modes and Responses

Testing for Partial Failure

Partial failure behaviour cannot be verified by testing the happy path. Failure must be injected deliberately.

Observing Partial Failure

Common Design Mistakes

System Design Interview Discussion

Design Checklist

Knowledge Check

Summary

Partial failure is the defining characteristic of distributed systems. Unlike a single process, where failure is total, distributed components fail independently, and the surviving components cannot determine which failed or whether requested work was completed.

A timeout conveys nothing about remote state. The two generals problem establishes that certain agreement over an unreliable network is impossible, which is why exactly-once delivery cannot exist and idempotency becomes foundational rather than optional.

Failure detection is always a guess balancing false positives against slow recovery. Gray failures, where components are slow or intermittently wrong, evade binary detectors and frequently cause more damage than clean crashes.

Partial failure becomes total failure through cascades, usually amplified by uncoordinated retries. Containment comes from bulkheads isolating resources, circuit breakers failing fast, deadline propagation bounding work, and deliberate degradation that sacrifices non-essential functionality rather than the whole service.