Prerequisites

The Dual-Write Problem

Almost every event-driven service needs to do two things when something happens: persist the change and tell other services about it. These are two separate systems, and there is no transaction spanning both.

// This code is wrong, and the reason is not obvious
async function placeOrder(order) {
    await database.orders.insert(order);          // Step one
    await broker.publish("OrderPlaced", order);   // Step two
}

Reversing the Order Does Not Help

// Also wrong, in the opposite direction
async function placeOrder(order) {
    await broker.publish("OrderPlaced", order);   // Now first
    await database.orders.insert(order);          // Now second
}

Why Retries Do Not Solve It

The Outbox Solution

The insight is simple: if the two writes must be atomic, make them the same write. Record the message in the same database, in the same transaction, then deliver it separately.

CREATE TABLE outbox (
    id              BIGSERIAL    PRIMARY KEY,
    aggregate_type  VARCHAR(80)  NOT NULL,
    aggregate_id    VARCHAR(80)  NOT NULL,
    event_type      VARCHAR(80)  NOT NULL,
    payload         JSONB        NOT NULL,
    headers         JSONB        NULL,
    created_at      TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP,
    published_at    TIMESTAMP    NULL,
    attempts        INT          NOT NULL DEFAULT 0
);

CREATE INDEX idx_outbox_unpublished
    ON outbox (id)
    WHERE published_at IS NULL;

BEGIN;

INSERT INTO orders (order_id, customer_id, total, status, created_at)
VALUES (:order_id, :customer_id, :total, 'placed', CURRENT_TIMESTAMP);

INSERT INTO outbox (
    aggregate_type, aggregate_id, event_type, payload
)
VALUES (
    'order',
    :order_id,
    'OrderPlaced',
    :event_payload
);

COMMIT;

async function placeOrder(database, order) {
    return await database.transaction(async (tx) => {
        const saved = await tx.orders.insert(order);

        await tx.outbox.insert({
            aggregateType: "order",
            aggregateId: saved.orderId,
            eventType: "OrderPlaced",
            payload: {
                orderId: saved.orderId,
                customerId: saved.customerId,
                items: saved.items,
                total: saved.total,
                placedAt: saved.createdAt
            }
        });

        return saved;
    });
}

The Message Relay

A separate process reads unpublished outbox rows and delivers them to the broker. Because the rows are durable, it can retry indefinitely without losing anything.

class OutboxRelay {
    async pollOnce() {
        const batch = await this.db.query(`
            SELECT id, aggregate_type, aggregate_id,
                   event_type, payload, headers
            FROM outbox
            WHERE published_at IS NULL
            ORDER BY id
            LIMIT $1
            FOR UPDATE SKIP LOCKED
        `, [this.batchSize]);

        if (batch.length === 0) return { published: 0 };

        let published = 0;

        for (const row of batch) {
            try {
                await this.broker.publish({
                    topic: this.topicFor(row.event_type),
                    key: row.aggregate_id,
                    value: row.payload,
                    headers: {
                        ...row.headers,
                        messageId: String(row.id),
                        eventType: row.event_type
                    }
                });

                await this.markPublished(row.id);
                published += 1;
            } catch (error) {
                await this.recordAttempt(row.id, error);
                break;
            }
        }

        return { published };
    }

    async markPublished(id) {
        await this.db.query(`
            UPDATE outbox
            SET published_at = CURRENT_TIMESTAMP
            WHERE id = $1
        `, [id]);
    }
}

Polling Versus Change Data Capture

The CDC Variant

-- With CDC, the row can be deleted in the same transaction
BEGIN;

INSERT INTO orders (order_id, customer_id, total, status)
VALUES (:order_id, :customer_id, :total, 'placed');

INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
VALUES ('order', :order_id, 'OrderPlaced', :payload);

DELETE FROM outbox WHERE aggregate_id = :order_id;

COMMIT;
-- The insert and delete both appear in the log; the connector
-- reads the insert and publishes, while the table stays empty

Ordering

The outbox preserves the order in which events were written, but only if the relay respects it and the broker partitions accordingly.

async function publishInOrder(rows, broker) {
    const byAggregate = new Map();

    for (const row of rows) {
        const key = row.aggregate_id;
        if (!byAggregate.has(key)) byAggregate.set(key, []);
        byAggregate.get(key).push(row);
    }

    // Different aggregates in parallel; within one, strictly sequential
    await Promise.all(
        Array.from(byAggregate.entries()).map(async ([key, group]) => {
            for (const row of group) {
                try {
                    await broker.publish({ key, value: row.payload });
                    await markPublished(row.id);
                } catch (error) {
                    return { key, stoppedAt: row.id, error };
                }
            }
        })
    );
}

Running Multiple Relay Instances

A single relay is a throughput ceiling and a single point of failure. Running several requires preventing them from publishing the same rows.

-- Shard assignment keeps each aggregate on one relay
SELECT id, aggregate_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
  AND MOD(ABS(HASHTEXT(aggregate_id)), :total_shards) = :my_shard
ORDER BY id
LIMIT :batch_size
FOR UPDATE SKIP LOCKED;

Cleanup and Growth

Published rows must be removed or the table grows without bound, degrading the very query the relay depends on.

-- Delete in bounded batches to avoid long locks
DELETE FROM outbox
WHERE id IN (
    SELECT id
    FROM outbox
    WHERE published_at IS NOT NULL
      AND published_at < CURRENT_TIMESTAMP - INTERVAL '7 days'
    ORDER BY id
    LIMIT 10000
);

The Inbox Pattern

The outbox guarantees at-least-once delivery. The receiving side needs a corresponding mechanism to make duplicate delivery harmless.

CREATE TABLE inbox (
    message_id   VARCHAR(120) PRIMARY KEY,
    consumer     VARCHAR(80)  NOT NULL,
    processed_at TIMESTAMP    NOT NULL DEFAULT CURRENT_TIMESTAMP
);

async function consumeMessage(database, message, handler) {
    return await database.transaction(async (tx) => {
        const claimed = await tx.query(`
            INSERT INTO inbox (message_id, consumer)
            VALUES ($1, $2)
            ON CONFLICT (message_id) DO NOTHING
            RETURNING message_id
        `, [message.id, handler.name]);

        if (claimed.rowCount === 0) {
            return { status: "duplicate-ignored" };
        }

        await handler.process(tx, message);

        return { status: "processed" };
    });
}

What to Put in the Payload

{
    "messageId": "48120",
    "eventType": "OrderPlaced",
    "aggregateType": "order",
    "aggregateId": "order-7734",
    "occurredAt": "2026-09-24T06:41:09Z",
    "schemaVersion": 2,
    "payload": {
        "orderId": "order-7734",
        "customerId": "customer-4821",
        "total": 249900,
        "currency": "INR",
        "items": [
            { "sku": "sku-501", "quantity": 2, "unitPrice": 124950 }
        ]
    },
    "traceId": "trace-9a2d5f"
}

Failure Modes

-- Poison message handling: bound the attempts
UPDATE outbox
SET attempts   = attempts + 1,
    last_error = :error_message
WHERE id = :id;

-- Route persistently failing rows aside
UPDATE outbox
SET published_at = CURRENT_TIMESTAMP,
    dead_lettered = TRUE
WHERE id = :id
  AND attempts >= :max_attempts;

Where the Outbox Fits

Costs and Alternatives

Monitoring

Verification

describe("transactional outbox", function () {
    it("writes no outbox row when the business write fails", async function () {
        await expect(
            placeOrder(database, invalidOrder)
        ).rejects.toThrow();

        const pending = await database.outbox.unpublished();
        expect(pending).toHaveLength(0);
    });

    it("retains the event when the broker is unavailable", async function () {
        await placeOrder(database, validOrder);
        await faultInjector.stopBroker();

        await relay.pollOnce();

        const pending = await database.outbox.unpublished();
        expect(pending).toHaveLength(1);
    });

    it("republishes after a crash between publish and mark", async function () {
        await placeOrder(database, validOrder);
        await faultInjector.crashAfterPublish();

        await relay.restart();
        await relay.pollOnce();

        expect(broker.messagesFor(validOrder.orderId)).toHaveLength(2);
    });

    it("processes a duplicate exactly once at the consumer", async function () {
        const message = buildMessage(validOrder);

        await consumeMessage(database, message, handler);
        await consumeMessage(database, message, handler);

        expect(handler.invocationCount).toBe(1);
    });

    it("preserves order per aggregate", async function () {
        await updateOrder(database, orderId, { status: "confirmed" });
        await updateOrder(database, orderId, { status: "shipped" });

        await relay.drain();

        const sequence = broker.messagesFor(orderId).map(m => m.status);
        expect(sequence).toEqual(["confirmed", "shipped"]);
    });
});

Common Design Mistakes

System Design Interview Discussion

Design Checklist

Knowledge Check

Summary

The dual-write problem arises whenever a service must persist a change and notify others about it. These are two independent systems with no shared transaction, so a crash between them produces either a lost event or a phantom one.

No ordering of the two writes avoids this, and retries do not help because the process itself can die. The transactional outbox resolves it by making both writes the same write: the message is inserted into a table in the same local transaction as the business change.

A separate relay then delivers those rows, either by polling or by tailing the database log. Because the rows are durable, delivery can be retried indefinitely. Ordering is preserved by reading sequentially and partitioning by aggregate identifier, and multiple relays can run if sharded by that same key.

The guarantee is at-least-once, not exactly-once, because the relay may publish and crash before recording success. The inbox pattern completes the picture, recording processed message identifiers in the consumer's own transaction so duplicates become harmless.