Introduction

Distributed systems communicate across networks that can delay, duplicate, interrupt, or lose requests and responses. A client can send a request, the server can complete the operation, and the response can disappear before reaching the client.

The client then faces an important question:

Did the server complete the operation,
or did the request fail before execution?

Retrying can recover from a temporary failure, but an unsafe retry can produce duplicate orders, payments, messages, refunds, reservations, or workflow transitions.

Idempotency allows a supported operation to be repeated without accumulating additional intended business effects beyond the first successful application.

In your System Design curriculum, Idempotency is Topic 4.7 under API Design & Service Contracts. It follows versioning and precedes authentication versus authorization and rate-limit semantics. The module's practical project includes a versioned REST API with cursor pagination and idempotency keys.

Prerequisites

What Is Idempotency?

An operation is idempotent when its intended server-side effect remains the same whether the identical operation is performed once or several times.

Mathematical Intuition

For an idempotent operation \(f\):

\[ f(f(x)) = f(x) \]

Applying the operation again does not create an additional intended effect beyond the result of the first application.

Set vs Increment

Idempotent operation:

Set order status to cancelled.

Apply once:
status = cancelled

Apply again:
status = cancelled


Non-idempotent operation:

Increase reward points by 100.

Apply once:
points increase by 100

Apply again:
points increase by another 100

Safety vs Idempotency

Safe and idempotent are related but different HTTP properties.

HTTP Method Semantics

Why PUT Is Idempotent

PUT /customers/cus_42/preferences HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
  "language": "en",
  "theme": "dark"
}

Sending the same replacement several times leaves the resource in the same intended state.

First request:

language = en
theme = dark


Identical retry:

language = en
theme = dark


Final intended state:

Unchanged by repetition

The service can still create separate logs, metrics, or audit records for each received request. Those incidental effects do not necessarily change the idempotency of the requested resource operation.

Why DELETE Is Idempotent

DELETE /customers/cus_42/payment-methods/pm_17 HTTP/1.1
Host: api.example.com

First request:

Payment method association is removed.
Response can be 204 No Content.


Second identical request:

Association is already absent.
Response can be 404 Not Found.


Intended final state:

The association does not exist.

Responses can differ while the intended effect remains the same.

Is PATCH Idempotent?

PATCH is not guaranteed to be idempotent. Its behaviour depends on the patch operation.

Naturally Idempotent Patch

{
  "status": "inactive"
}

Repeatedly setting the same field to the same value produces the same intended final state.

Non-idempotent Patch

{
  "operation": "increment",
  "field": "rewardPoints",
  "amount": 100
}

Every retry adds another 100 points. This operation needs explicit idempotency protection when retries are supported.

The Uncertain Outcome Problem

Client sends payment request
        |
        v
Server validates request
        |
        v
Server creates payment
        |
        v
Database transaction commits
        |
        v
Response is lost
        |
        v
Client receives timeout


Question:

Did the payment fail,
or did it succeed without a visible response?

The timeout describes what the client observed. It does not prove whether the server applied the operation.

Idempotency-key Pattern

An idempotency key is a client-generated value identifying one logical operation. The client sends the same key whenever the same operation is retried.

POST /payments HTTP/1.1
Host: api.example.com
Content-Type: application/json
Idempotency-Key: 1b1ad8f8-4707-4e28-8b54-46ec06163a23

{
  "orderId": "ord_9001",
  "amount": {
    "value": "149.50",
    "currency": "USD"
  }
}

The server associates the key with the caller, operation, normalized request fingerprint, processing state, and stored outcome.

Client Responsibilities

A client using idempotency keys should:

• Generate a unique key for each new logical operation
• Reuse the same key only when retrying that operation
• Send the same semantically relevant request content on every retry
• Keep retries within the server's documented key-retention period
• Use bounded retries with backoff and jitter
• Stop retrying after the complete operation deadline
• Handle an in-progress response according to the contract
• Treat a key-conflict response as a request error

Key: checkout-123

Request 1:
Charge USD 100

Request 2:
Charge USD 250

Key: payment-1b1ad8f8

Original request:
Charge order ord_9001 for USD 149.50

Retry:
Same key
Same authenticated caller
Same operation
Same relevant request content

Server Responsibilities

A server supporting idempotency keys should define:

• Which operations accept or require a key
• The header or field used to carry the key
• The accepted key format
• The minimum and maximum key length
• The caller or tenant scope
• The request fingerprint rules
• Concurrent-request behaviour
• Which outcomes are stored
• The key-retention period
• The response after key expiration

Request Fingerprint

The server should verify that reuse of an idempotency key refers to the same operation and semantically relevant request content.

Fingerprint inputs can include:

- Authenticated tenant
- Authenticated caller
- HTTP method
- Normalized route
- Relevant request body
- Currency and amount
- Target resource
- Operation type

Conceptual Fingerprint

fingerprint =
SHA-256(
  tenant_id
  + operation_name
  + canonical_request_content
)

Raw JSON text is not always an appropriate fingerprint because harmless whitespace or property-order changes can produce different byte strings. The contract should define canonicalization or select business fields explicitly.

Idempotency Record

Example SQL Table

CREATE TABLE idempotency_records
(
    tenant_id VARCHAR(64) NOT NULL,
    idempotency_key VARCHAR(128) NOT NULL,
    operation_name VARCHAR(100) NOT NULL,
    request_fingerprint CHAR(64) NOT NULL,
    processing_status VARCHAR(20) NOT NULL,
    resource_id VARCHAR(64) NULL,
    response_status INT NULL,
    response_body TEXT NULL,
    created_at DATETIME NOT NULL,
    completed_at DATETIME NULL,
    expires_at DATETIME NOT NULL,

    PRIMARY KEY
    (
        tenant_id,
        idempotency_key
    ),

    INDEX ix_idempotency_expiry
    (
        expires_at
    )
);

The schema is illustrative. The final structure depends on database capabilities, payload size, retention requirements, encryption policy, and whether complete responses or only resource references are stored.

Idempotency Record States

ABSENT
  |
  | Reserve key successfully
  v
IN_PROGRESS
  |
  +--> Business operation succeeds
  |       |
  |       v
  |    COMPLETED
  |
  +--> Validation fails before execution
  |       |
  |       v
  |    Record can be released or stored
  |
  +--> Execution fails
          |
          v
       FAILED according to policy

The in-progress state is essential. It prevents two concurrent requests with the same key from both observing that the key is absent and executing the operation independently.

Concurrent Duplicate Requests

Request A with key K
        |
        +--> Checks K: absent

Request B with key K
        |
        +--> Checks K: absent

Without atomic reservation:

Request A executes payment.
Request B executes payment.

Result:

Duplicate business effect.

Atomic Reservation

Request A:

Atomically inserts key K.
Insert succeeds.
Request A becomes the owner.


Request B:

Attempts to insert key K.
Unique constraint rejects duplicate.
Request B loads existing record.


Existing state:

IN_PROGRESS:
Return or wait according to policy.

COMPLETED:
Replay stored outcome.

Handling an In-progress Retry

When a duplicate request arrives while the original request is still processing, the API needs an explicit response policy.

Possible policies include:

• Return a conflict indicating that the operation is already in progress
• Return an accepted response with an operation-status resource
• Wait for the original operation within a bounded deadline
• Return a retry instruction defined by the API contract

In-progress Response

HTTP/1.1 409 Conflict
Content-Type: application/problem+json
Retry-After: 2

{
  "type": "idempotency-request-in-progress",
  "title": "The operation is already in progress",
  "status": 409,
  "detail": "Retry the same request using the same idempotency key."
}

The selected status and retry behaviour should be documented consistently.

Replaying a Completed Outcome

Retry arrives with key K
        |
        v
Existing record is COMPLETED
        |
        v
Fingerprint matches
        |
        v
Return stored status and representation
        |
        v
Do not execute the business operation again

Replayed Response

HTTP/1.1 201 Created
Location: /payments/pay_7001
Content-Type: application/json
Idempotency-Replayed: true

{
  "id": "pay_7001",
  "orderId": "ord_9001",
  "status": "authorized",
  "amount": {
    "value": "149.50",
    "currency": "USD"
  }
}

A replay indicator is an API-specific convenience and should be documented when provided.

Same Key, Different Request

Reusing a key with different request content is a client error.

HTTP/1.1 409 Conflict
Content-Type: application/problem+json

{
  "type": "idempotency-key-conflict",
  "title": "Idempotency key conflict",
  "status": 409,
  "detail": "The key was previously used with different request content."
}

The server must not silently return an unrelated stored response or execute a new operation under the same scoped key.

Coordinate Business State and Idempotency State

The business operation and idempotency record must not become inconsistent.

Unsafe sequence:

1. Create payment.
2. Commit payment.
3. Attempt to store idempotency result.
4. Process fails before step 3 finishes.

Retry:

No completed idempotency result is found.
Payment can be created again.

Transactional Sequence

Begin transaction

1. Reserve or lock idempotency record.
2. Validate fingerprint.
3. Create business resource.
4. Mark idempotency record completed.
5. Store resource reference or response.

Commit transaction

When the business resource and idempotency record are stored in different systems, a single local transaction may not be available. The architecture then needs a recoverable workflow, durable operation state, or another distributed consistency strategy.

What Should Be Stored?

The API contract should define whether a replay returns the original stored response or a newly generated representation of the existing resource.

Key Retention and Expiration

Idempotency records cannot normally be stored forever. The service should define a retention period based on the maximum retry window and business risk.

Key created
    |
    v
Active retry window
    |
    v
Key expires
    |
    v
Record becomes eligible for cleanup

The contract should define:

• How long a key remains recognized
• Whether clients can retry after expiration
• Whether expired keys can be reused
• How asynchronous operations affect retention
• How cleanup is performed

Cleanup Query

DELETE FROM idempotency_records
WHERE
    expires_at < CURRENT_TIMESTAMP
    AND processing_status IN
    (
        'COMPLETED',
        'FAILED'
    );

Cleanup should normally run in bounded batches to prevent large deletes, extended locks, or excessive transaction-log growth.

Scope the Key to the Caller

Idempotency keys should be scoped to a trusted identity, tenant, account, or operation namespace.

Primary key:

idempotency_key


Problem:

Two unrelated customers can accidentally
or intentionally submit the same key.

Unique key:

tenant_id
+
idempotency_key

The tenant or caller identity must come from trusted authentication context, not from an unverified request-body field.

Authorization Still Applies

An idempotency record does not replace authorization. Every original request and retry must be evaluated within the correct security context.

Retry with existing key
        |
        v
Authenticate caller
        |
        v
Confirm key belongs to caller's scope
        |
        v
Verify permission for operation or result
        |
        v
Return authorized stored outcome

A caller must not obtain another user's result merely by discovering or guessing an idempotency key.

Retry Policy

Idempotency enables selected retries, but clients still need a bounded retry policy.

Call fails
    |
    v
Is failure retryable?
    |
    +--> No:
    |       return failure
    |
    +--> Yes:
            Is operation idempotent
            or protected by a valid key?
                |
                +--> No:
                |       do not retry automatically
                |
                +--> Yes:
                        wait using backoff and jitter
                        retry within overall deadline

A retry policy should define:

• Retryable failures
• Maximum attempts
• Complete operation deadline
• Exponential backoff
• Randomized jitter
• Idempotency-key reuse
• Cancellation handling

Exponential Backoff

A simplified backoff formula is:

\[ Delay_n = \min \left( MaximumDelay, BaseDelay \times 2^n \right) + Jitter \]

Jitter reduces the chance that many clients retry simultaneously after a shared outage.

Payment Example

Logical operation:

Authorize USD 149.50
for order ord_9001.


Client generates:

Idempotency key K.


Request 1:

Server creates payment pay_7001.
Response is lost.


Request 2 with key K:

Server finds completed record.
Server returns pay_7001.
No second payment is created.

The payment provider can also require its own idempotency identifier. The application should send a stable downstream key derived from the logical operation instead of generating a new downstream key for every internal retry.

Idempotency across Service Boundaries

Client
  |
  | Key: K
  v
Order API
  |
  | Stable downstream operation ID
  v
Payment Service
  |
  | Stable provider key
  v
Payment Provider

Every boundary that can repeat the operation needs its own deduplication or idempotency strategy. Protecting only the public API does not automatically protect repeated internal messages or downstream calls.

Idempotent Message Consumers

Message brokers and webhook providers can deliver the same event more than once. Consumers should use a durable event identifier to prevent duplicate effects.

Event evt_7001 arrives
        |
        v
Insert evt_7001 into processed-events table
        |
        +--> Insert succeeds:
        |       process event once
        |
        +--> Duplicate constraint:
                event was already processed
                acknowledge without repeating effect

Processed-event Table

CREATE TABLE processed_events
(
    consumer_name VARCHAR(100) NOT NULL,
    event_id VARCHAR(128) NOT NULL,
    processed_at DATETIME NOT NULL,

    PRIMARY KEY
    (
        consumer_name,
        event_id
    )
);

The event reservation and business change should be coordinated transactionally where possible.

Idempotent Webhook Processing

Webhook received
      |
      v
Validate signature and timestamp
      |
      v
Extract trusted event ID
      |
      v
Reserve event ID atomically
      |
      +--> New event:
      |       apply business effect
      |
      +--> Duplicate event:
              do not repeat effect
      |
      v
Return acknowledgement

Idempotency does not replace webhook authentication. The receiver must first verify that the event came from the expected source.

GraphQL Mutation Idempotency

input CreateOrderInput {
  idempotencyKey: String!
  customerId: ID!
  lines: [CreateOrderLineInput!]!
}

type CreateOrderPayload {
  order: Order
  errors: [UserError!]!
}

type Mutation {
  createOrder(
    input: CreateOrderInput!
  ): CreateOrderPayload!
}

A retried mutation should reuse the same idempotency key and semantically identical input.

gRPC Idempotency

message CreatePaymentRequest {
  string idempotency_key = 1;
  string order_id = 2;
  Money amount = 3;
}

message CreatePaymentResponse {
  Payment payment = 1;
}

message Money {
  string value = 1;
  string currency = 2;
}

The gRPC service should document which methods can be retried and how keys are scoped, retained, and validated. A deadline expiration does not prove that the remote method did not complete.

PHP Idempotency-key Validation

<?php

declare(strict_types=1);

function readIdempotencyKey(): string
{
    $key =
        $_SERVER[
            'HTTP_IDEMPOTENCY_KEY'
        ] ?? '';

    $key =
        trim(
            $key
        );

    if ($key === '') {
        throw new InvalidArgumentException(
            'An idempotency key is required.'
        );
    }

    if (strlen($key) > 128) {
        throw new InvalidArgumentException(
            'The idempotency key is too long.'
        );
    }

    if (
        preg_match(
            '/^[A-Za-z0-9._:-]+$/',
            $key
        ) !== 1
    ) {
        throw new InvalidArgumentException(
            'The idempotency key format is invalid.'
        );
    }

    return $key;
}

The accepted format is application-defined. Publish the exact format, maximum length, and retention policy in the API contract.

Create a Request Fingerprint

<?php

declare(strict_types=1);

function createRequestFingerprint(
    string $tenantId,
    string $operationName,
    array $request
): string {
    $fingerprintData = [
        'tenantId' => $tenantId,
        'operation' => $operationName,
        'orderId' =>
            (string)($request['orderId'] ?? ''),
        'amount' =>
            (string)(
                $request['amount']['value'] ??
                ''
            ),
        'currency' =>
            strtoupper(
                (string)(
                    $request['amount']['currency'] ??
                    ''
                )
            )
    ];

    $canonical =
        json_encode(
            $fingerprintData,
            JSON_THROW_ON_ERROR |
            JSON_UNESCAPED_SLASHES
        );

    return hash(
        'sha256',
        $canonical
    );
}

Include only fields that define the logical operation. Do not include volatile transport metadata such as a retry timestamp unless the contract makes that value semantically relevant.

PHP Processing Flow

<?php

declare(strict_types=1);

function createPayment(
    PDO $pdo,
    string $tenantId,
    string $idempotencyKey,
    string $fingerprint,
    array $request
): array {
    $pdo->beginTransaction();

    try {
        $insert =
            $pdo->prepare(
                '
                INSERT INTO idempotency_records
                (
                    tenant_id,
                    idempotency_key,
                    operation_name,
                    request_fingerprint,
                    processing_status,
                    created_at,
                    expires_at
                )
                VALUES
                (
                    :tenant_id,
                    :idempotency_key,
                    :operation_name,
                    :request_fingerprint,
                    :processing_status,
                    CURRENT_TIMESTAMP,
                    DATE_ADD(
                        CURRENT_TIMESTAMP,
                        INTERVAL 1 DAY
                    )
                )
                '
            );

        try {
            $insert->execute([
                'tenant_id' => $tenantId,
                'idempotency_key' =>
                    $idempotencyKey,
                'operation_name' =>
                    'CreatePayment',
                'request_fingerprint' =>
                    $fingerprint,
                'processing_status' =>
                    'IN_PROGRESS'
            ]);
        } catch (PDOException $exception) {
            $pdo->rollBack();

            return loadExistingOutcome(
                $pdo,
                $tenantId,
                $idempotencyKey,
                $fingerprint
            );
        }

        $payment =
            insertPayment(
                $pdo,
                $tenantId,
                $request
            );

        $responseBody =
            json_encode(
                $payment,
                JSON_THROW_ON_ERROR
            );

        $complete =
            $pdo->prepare(
                '
                UPDATE idempotency_records
                SET
                    processing_status =
                        :processing_status,
                    resource_id =
                        :resource_id,
                    response_status =
                        :response_status,
                    response_body =
                        :response_body,
                    completed_at =
                        CURRENT_TIMESTAMP
                WHERE
                    tenant_id =
                        :tenant_id
                    AND idempotency_key =
                        :idempotency_key
                '
            );

        $complete->execute([
            'processing_status' =>
                'COMPLETED',
            'resource_id' =>
                $payment['id'],
            'response_status' => 201,
            'response_body' =>
                $responseBody,
            'tenant_id' => $tenantId,
            'idempotency_key' =>
                $idempotencyKey
        ]);

        $pdo->commit();

        return [
            'status' => 201,
            'body' => $payment,
            'replayed' => false
        ];
    } catch (Throwable $exception) {
        if ($pdo->inTransaction()) {
            $pdo->rollBack();
        }

        throw $exception;
    }
}

The example demonstrates the intended transaction shape. Production code should distinguish duplicate-key failures from unrelated database errors, implement the existing-outcome function safely, protect sensitive response data, and follow the selected database's transaction semantics.

Test Idempotency with curl

First Request

curl -i \
  -X POST \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: payment-test-1042' \
  --data '{
    "orderId": "ord_9001",
    "amount": {
      "value": "149.50",
      "currency": "USD"
    }
  }' \
  https://api.example.com/payments

Identical Retry

curl -i \
  -X POST \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: payment-test-1042' \
  --data '{
    "orderId": "ord_9001",
    "amount": {
      "value": "149.50",
      "currency": "USD"
    }
  }' \
  https://api.example.com/payments

Conflicting Reuse

curl -i \
  -X POST \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: payment-test-1042' \
  --data '{
    "orderId": "ord_9001",
    "amount": {
      "value": "250.00",
      "currency": "USD"
    }
  }' \
  https://api.example.com/payments

The first and identical retry should identify the same logical payment. The conflicting request should be rejected through the documented error contract.

Idempotency Observability

Useful idempotency metrics include:

• New idempotency keys
• Completed-key replays
• In-progress duplicate requests
• Fingerprint conflicts
• Key reservation failures
• Processing duration
• Expired-key cleanup volume
• Duplicate business-effect detection
• Retry count by operation
• Stored-response size

Logs should include an approved hash or safely handled form of the idempotency key, operation name, caller scope, state, and trace identifier. Sensitive request or response content should be redacted.

Idempotency Troubleshooting Workflow

1. Confirm the logical operation and endpoint.
2. Confirm whether the operation supports or requires idempotency.
3. Record the caller scope and key safely.
4. Compare the original and retry request fingerprints.
5. Inspect the idempotency record state.
6. Check whether concurrent requests used the same key.
7. Verify atomic reservation through a unique constraint.
8. Check whether the business transaction committed.
9. Inspect the stored resource identifier or response.
10. Check downstream retries and provider keys.
11. Verify key retention and cleanup behaviour.
12. Confirm that no duplicate business resource was created.

Common Idempotency Mistakes

Recommended Test Cases

Idempotency Best Practices

Practice Exercise

Implement idempotent order creation and payment processing in the versioned API developed during the earlier lessons.

Requirements

1. Require an idempotency key for order and payment creation.
2. Limit the accepted key length and characters.
3. Scope the key to the authenticated tenant.
4. Create a fingerprint from semantically relevant request fields.
5. Reserve the key using a unique database constraint.
6. Represent in-progress and completed states.
7. Store the created resource identifier and response status.
8. Return the stored result for an identical retry.
9. Reject the same key with different request content.
10. Handle two concurrent requests using the same key.
11. Coordinate order creation and key completion in one transaction.
12. Propagate a stable key to the payment provider.
13. Define a retention and cleanup policy.
14. Add replay, conflict, and processing-duration metrics.
15. Test response loss after transaction commit.

Design Template

Frequently Asked Questions