Prerequisites

Why Exactly-Once Delivery Cannot Exist

This is not an engineering limitation awaiting a better implementation. It follows directly from the two generals problem, which establishes that certain agreement over an unreliable channel is impossible.

Delivery Versus Processing

The confusion almost always stems from conflating two different things: how many times a message arrives, and how many times its effect is applied.

What Vendors Actually Guarantee

Exactly-once features are real and useful. They are also scoped far more narrowly than the marketing suggests.

How Platform Exactly-Once Works

Where Claims Break Down

// The guarantee covers the offset commit and the output topic.
// It does not cover the payment call.
async function handleOrderPlaced(message, context) {
    const order = message.value;

    // Outside any transaction the platform controls
    await paymentGateway.charge({
        customerId: order.customerId,
        amount: order.total
    });

    // Covered by the platform transaction
    await context.produce("OrderCharged", { orderId: order.orderId });

    // Offset commits atomically with the produce, not with the charge
}

Non-Determinism Breaks Replay

Exactly-once processing guarantees frequently assume that reprocessing the same input produces the same output. Ordinary code violates this constantly.

// Non-deterministic: replay produces a different record
function buildRecord(event) {
    return {
        id: crypto.randomUUID(),
        processedAt: new Date().toISOString(),
        orderId: event.orderId
    };
}

// Deterministic: replay produces an identical record
function buildRecord(event) {
    return {
        id: deriveId(event.orderId, event.eventId),
        processedAt: event.occurredAt,
        orderId: event.orderId
    };
}

Building Exactly-Once Effects

Since the platform guarantee stops at its boundary, the application must extend it. Three techniques cover most cases.

CREATE TABLE processed_messages (
    consumer     VARCHAR(80)  NOT NULL,
    message_id   VARCHAR(120) NOT NULL,
    processed_at TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (consumer, message_id)
);

CREATE INDEX idx_processed_cleanup
    ON processed_messages (processed_at);

async function processExactlyOnce(db, message, handler) {
    return await db.transaction(async (tx) => {
        const claim = await tx.query(`
            INSERT INTO processed_messages (consumer, message_id)
            VALUES ($1, $2)
            ON CONFLICT (consumer, message_id) DO NOTHING
            RETURNING message_id
        `, [handler.name, message.id]);

        if (claim.rowCount === 0) {
            return { status: "duplicate", messageId: message.id };
        }

        // Effect and deduplication record share one commit
        await handler.apply(tx, message);

        return { status: "applied", messageId: message.id };
    });
}

When the Effect Is External

async function chargeExactlyOnce(db, message, gateway) {
    const idempotencyKey = `charge:${message.orderId}:${message.eventId}`;

    // The gateway deduplicates; we do not need to
    const result = await gateway.charge({
        amount: message.total,
        customerId: message.customerId,
        idempotencyKey
    });

    await db.query(`
        INSERT INTO charge_records (order_id, gateway_ref, idempotency_key)
        VALUES ($1, $2, $3)
        ON CONFLICT (idempotency_key) DO NOTHING
    `, [message.orderId, result.reference, idempotencyKey]);

    return result;
}

Deduplication Windows

Deduplication state cannot be retained forever. Every implementation has a window, and duplicates arriving after it are not recognized.

-- Retain well beyond the longest plausible redelivery
DELETE FROM processed_messages
WHERE processed_at < CURRENT_TIMESTAMP - INTERVAL '30 days'
  AND ctid IN (
      SELECT ctid FROM processed_messages
      WHERE processed_at < CURRENT_TIMESTAMP - INTERVAL '30 days'
      LIMIT 10000
  );

The Cost of Exactly-Once

Interrogating a Claim

When a platform advertises exactly-once, these questions reveal what is actually covered.

A Worked Example

An order pipeline where each step needs a different guarantee.

Monitoring

Verification

describe("exactly-once effects", function () {
    it("applies an effect once despite duplicate delivery", async function () {
        const message = buildMessage({ orderId: "o-1", points: 100 });

        await processExactlyOnce(db, message, loyaltyHandler);
        await processExactlyOnce(db, message, loyaltyHandler);
        await processExactlyOnce(db, message, loyaltyHandler);

        const balance = await db.loyalty.balanceFor("customer-1");
        expect(balance).toBe(100);
    });

    it("does not record the message when the effect fails", async function () {
        loyaltyHandler.failNext();

        await expect(
            processExactlyOnce(db, message, loyaltyHandler)
        ).rejects.toThrow();

        const recorded = await db.processedMessages.exists(message.id);
        expect(recorded).toBe(false);
    });

    it("reprocesses after the deduplication window expires", async function () {
        await processExactlyOnce(db, message, loyaltyHandler);
        await db.processedMessages.expireAll();

        await processExactlyOnce(db, message, loyaltyHandler);

        const balance = await db.loyalty.balanceFor("customer-1");
        expect(balance).toBe(200); // Documents the real limitation
    });

    it("produces identical output on replay", async function () {
        const first = buildRecord(message);
        const second = buildRecord(message);

        expect(first).toEqual(second);
    });

    it("passes a stable idempotency key to the gateway", async function () {
        await chargeExactlyOnce(db, message, gateway);
        await chargeExactlyOnce(db, message, gateway);

        expect(gateway.distinctKeysReceived()).toBe(1);
        expect(gateway.chargesApplied()).toBe(1);
    });
});

Common Design Mistakes

System Design Interview Discussion

Design Checklist

Knowledge Check

Summary

Exactly-once delivery is provably impossible. A sender that receives no acknowledgement cannot determine whether the message or the acknowledgement was lost, so it must choose between risking duplication and risking loss.

What is achievable is exactly-once effect: at-least-once delivery combined with deduplication somewhere in the path. Every real exactly-once feature works this way, using producer sequence numbers to discard retries and committing output atomically with the consumer offset.

Those guarantees are genuine but narrowly scoped. They hold within the platform boundary and end at the first external side effect. They also typically assume deterministic processing, which ordinary code violates through clocks, random identifiers, and external calls.

The practical response is to assume at-least-once everywhere, classify each effect as duplicate-safe or not, and extend the guarantee only where it matters through deduplication tables committed with the effect or idempotency keys passed downstream.