Prerequisites

What Counts as a Secret

A secret is any value that grants access or proves identity, and whose disclosure would allow an unauthorized party to act as a legitimate component of the system.

Secret Sprawl

Secrets multiply quietly. A value created once for convenience gets copied into a second environment, pasted into a message, and referenced by a script, until nobody can enumerate where it exists.

The practical cost of rotating a secret scales with the number of places it was copied to. This is why sprawl is not merely untidy: it is the reason rotation gets deferred indefinitely.

Secret Management Architecture

A secret manager centralizes storage, access control, auditing, and rotation. Applications request secrets at startup or on demand rather than receiving them through deployment artifacts.

The Bootstrap Problem

If secrets live in a manager, the application needs a credential to authenticate to that manager. Storing that credential in configuration simply relocates the original problem.

Static Versus Dynamic Secrets

Short leases bound this window automatically. A credential valid for an hour limits damage even when compromise goes undetected, which is the common case.

Why Rotation Matters

Rotation replaces a credential with a new value and invalidates the old one. It serves several distinct purposes that are often conflated.

The Dual-Credential Pattern

Naive rotation causes an outage: the old credential is revoked before every consumer has picked up the new one. The dual pattern eliminates that gap by allowing two valid credentials during the transition.

CREATE TABLE managed_secrets (
    secret_id        VARCHAR(80)  PRIMARY KEY,
    secret_name      VARCHAR(160) NOT NULL,
    secret_type      VARCHAR(40)  NOT NULL,
    current_version  INT          NOT NULL,
    previous_version INT          NULL,
    owner_team       VARCHAR(80)  NOT NULL,
    rotation_days    INT          NOT NULL,
    last_rotated_at  TIMESTAMP    NOT NULL,
    next_rotation_at TIMESTAMP    NOT NULL,
    last_verified_at TIMESTAMP    NULL
);

SELECT secret_name,
       secret_type,
       owner_team,
       last_rotated_at,
       next_rotation_at
FROM managed_secrets
WHERE next_rotation_at < CURRENT_TIMESTAMP
ORDER BY next_rotation_at;

Orchestrating the Rotation

async function rotateSecret(secretName) {
    const record = await secretStore.getMetadata(secretName);

    const newValue = await provider.createCredential({
        secretName: secretName,
        keepExisting: true
    });

    const newVersion = await secretStore.addVersion(
        secretName,
        newValue,
        { stage: "current" }
    );

    await secretStore.setVersionStage(
        secretName,
        record.currentVersion,
        "previous"
    );

    await notifyConsumers(secretName, newVersion);

    const migrated = await waitForMigration(secretName, {
        timeoutMs: record.propagationWindowMs
    });

    if (!migrated.complete) {
        await alerting.raise({
            severity: "high",
            message: `Rotation stalled for ${secretName}`,
            stillUsingOld: migrated.laggingConsumers
        });
        return { status: "PENDING_MIGRATION" };
    }

    await provider.disableCredential(record.currentVersion);

    await secretStore.updateMetadata(secretName, {
        previousVersion: record.currentVersion,
        currentVersion: newVersion,
        lastRotatedAt: new Date().toISOString()
    });

    return { status: "ROTATED", version: newVersion };
}

Rotation by Secret Type

Signing Key Overlap

{
    "keys": [
        {
            "kid": "signing-2026-09",
            "status": "active",
            "use": "sig",
            "activatedAt": "2026-09-01T00:00:00Z"
        },
        {
            "kid": "signing-2026-06",
            "status": "deprecated",
            "use": "sig",
            "acceptUntil": "2026-09-30T00:00:00Z"
        }
    ]
}

Because tokens signed by the previous key remain in circulation until they expire, the overlap window must exceed the maximum token lifetime. Removing the old key sooner invalidates valid tokens.

Connection Pools and Cached Credentials

Long-lived connections are the most frequent cause of rotation failures. A pool established at startup keeps using the old credential until connections are recycled.

class CredentialProvider {
    constructor(secretName, options) {
        this.secretName = secretName;
        this.ttlMs = options.ttlMs;
        this.cached = null;
        this.fetchedAt = 0;
    }

    async get({ forceRefresh = false } = {}) {
        const age = Date.now() - this.fetchedAt;

        if (!forceRefresh && this.cached && age < this.ttlMs) {
            return this.cached;
        }

        this.cached = await secretManager.getCurrent(this.secretName);
        this.fetchedAt = Date.now();

        return this.cached;
    }
}

async function queryWithRetry(sql, params) {
    try {
        return await database.query(sql, params);
    } catch (error) {
        if (!isAuthenticationError(error)) {
            throw error;
        }

        await credentialProvider.get({ forceRefresh: true });
        await database.reconnect();

        return await database.query(sql, params);
    }
}

Blast Radius

The impact of a compromised secret depends on how much it unlocks and how many systems share it. Partitioning credentials limits what a single leak achieves.

Detecting Leaked Secrets

Secrets leak through routine developer activity far more often than through sophisticated attack. Detection must operate continuously rather than at review time.

Emergency Rotation

Scheduled rotation prioritizes availability. Emergency rotation prioritizes containment, and accepts disruption as the cost of closing exposure quickly.

Threats and Mitigations

Monitoring and Auditing

Common Design Mistakes

System Design Interview Discussion

Implementation Checklist

Knowledge Check

Summary

Secrets are the credentials that let components prove identity and obtain access. They accumulate copies across repositories, images, pipelines, and machines, and that sprawl is the reason rotation gets postponed.

A secret manager provides one authoritative source with encrypted storage, identity-based access, versioning, and auditing. Platform-issued workload identity resolves the bootstrap problem without writing a credential into any artifact.

Rotation bounds undetected compromise and, equally importantly, keeps the procedure exercised. The dual-credential pattern avoids outages by allowing two valid values, but only if the old value is revoked after verified migration rather than on a timer.

Connection pools and cached credentials are the usual cause of rotation failures. Narrow blast radius through per-workload credentials and strict environment isolation, scan continuously for leaks, and treat any exposure as grounds for immediate rotation.