Introduction

Replication maintains copies of data on multiple machines. These copies can improve availability, read capacity, fault tolerance, and geographic access.

However, a data change cannot appear on every replica instantaneously. The change must be transferred across a network, received by each replica, persisted according to the database policy, and applied to the replica's local state.

Client writes new data
      |
      v
Leader commits the write
      |
      v
Replication record is transmitted
      |
      v
Follower receives the record
      |
      v
Follower persists and applies it
      |
      v
Follower can return the new value

The delay between the authoritative write and its visibility on another replica is called replication lag.

Prerequisites

What Is Replication Lag?

Replication lag is the difference between the latest accepted state of a dataset and the state currently available on another replica.

Leader:

Latest replication position = 15,800


Follower A:

Applied position = 15,800


Follower B:

Applied position = 15,720


Follower C:

Applied position = 15,100

Follower A is current at the measured position. Followers B and C remain behind and can return older values for changes they have not yet applied.

Lag can be represented using:

• Time behind the source
• Bytes of unapplied replication data
• Log-sequence position difference
• Number of pending records or operations
• Transaction or commit position difference
• Oldest unapplied event age

Replication-lag Stages

A replica can have received a change without having made it visible to ordinary reads. Monitor the stage that matches the database's actual replication architecture.

Synchronous vs Asynchronous Replication

Synchronous Replication

Client sends write
      |
      v
Leader commits or prepares write
      |
      v
Required follower acknowledges
      |
      v
Leader returns success

Synchronous replication places a configured replica acknowledgment in the client write path. It can provide stronger durability or visibility guarantees, depending on what the acknowledgment means.

The trade-offs can include:

• Higher write latency
• Dependence on replica and network response time
• Reduced write availability when required replicas are unavailable

Asynchronous Replication

Client sends write
      |
      v
Leader commits write
      |
      v
Leader returns success
      |
      v
Followers receive and apply
the change later

Asynchronous replication allows the leader to acknowledge a write before followers confirm it. This can reduce write latency and preserve leader availability during follower problems, but followers can return stale data.

Lag across Replication Topologies

Leader-Follower

Writes
  |
  v
Leader
  |
  +-- Replication stream -> Follower A
  +-- Replication stream -> Follower B
  +-- Replication stream -> Follower C

Followers can be at different replication positions. A read from a lagging follower can omit recent leader writes.

Multi-Leader

Region A Leader
      |
      | cross-region replication
      v
Region B Leader


Region B Leader
      |
      | cross-region replication
      v
Region A Leader

Each leader can accept local writes before remote leaders receive them. Cross-location lag can therefore combine stale reads with concurrent-write conflicts.

Leaderless

Coordinator
   |
   +-- Replica A: Version 12
   +-- Replica B: Version 12
   +-- Replica C: Version 10

Leaderless systems can observe stale or concurrent versions across replicas. Quorum reads, version comparison, read repair, hinted handoff, and anti-entropy are used according to the database's design.

User-visible Stale Read

1. User updates course progress.

2. Write is committed on the leader.

3. Application returns success.

4. User immediately refreshes the page.

5. Read is sent to a lagging follower.

6. Follower returns the previous progress value.

The database can be operating according to its asynchronous replication contract while the application still appears incorrect to the user.

Read-Your-Writes Consistency

Read-your-writes consistency means that after a client successfully writes a value, that client will not subsequently read a version older than the accepted write.

Client writes:

Progress = 72%


Required subsequent observation:

Progress >= Version containing 72%

Possible implementation patterns include:

• Read from the leader after a client write
• Read from the leader for data owned or modified by the current user
• Track the write's replication position and select a replica that has reached it
• Wait for an eligible replica to catch up within a bounded deadline
• Use a database-provided consistency level that meets the requirement

Position-based Read Routing

The application can capture a replication position associated with a successful write and require subsequent reads to use a replica that has applied at least that position.

Write commits at:

Position 8,500


Subsequent read requires:

Applied position >= 8,500


Replica A:

Position 8,420
Not eligible


Replica B:

Position 8,510
Eligible

Conceptual Client Context

{
  "resourceId": "course-progress-1042-42",
  "minimumRequiredVersion": 8500
}

The version or position can remain opaque to the public client. The server should validate any client-carried consistency context rather than trusting an arbitrary client-selected database position.

Freshness-aware Read Router

Read request
      |
      v
Does request require
a minimum version?
      |
      +-- No:
      |      route to an eligible replica
      |
      +-- Yes:
             compare required version
             with replica positions
                    |
                    +-- Current replica exists:
                    |      route to replica
                    |
                    +-- No replica caught up:
                           route to leader,
                           wait within deadline,
                           or fail according
                           to the contract

Monotonic Reads

Monotonic reads ensure that after a client has observed one version, later reads do not return an older version.

First read from Follower A:

Course progress = Version 14


Second read from Follower B:

Course progress = Version 11


Problem:

The client observes data
moving backward.

Possible directions include:

• Route one client to a replica that is at least as current as the previous read
• Carry a minimum observed version in trusted session context
• Use affinity to an eligible replica where appropriate
• Route to the leader when no follower satisfies the minimum version

Consistent-prefix Reads

Consistent-prefix reads preserve a meaningful order between causally related writes.

Write 1:

Create discussion question


Write 2:

Create reply to that question


Invalid observation:

Reply appears before
the original question

Related data can be routed through the same partition, replication stream, or ordering mechanism when causal ordering must remain visible.

Three Important Guarantees

Replication Lag and Failover

If the leader fails, the system can promote a follower. A lagging follower can be missing recent writes acknowledged by the old leader.

Old leader:

Committed through Position 10,000


Follower selected for promotion:

Applied through Position 9,970


Potential missing range:

Positions 9,971 to 10,000

The exact outcome depends on replication acknowledgment, failover, durable logging, election, and recovery rules.

Failover design should define:

• Which followers are eligible for promotion
• How replication freshness is compared
• Whether acknowledged writes can be lost
• How the old leader is fenced
• How divergent or missing writes are handled
• When traffic can resume safely

Common Causes of Replication Lag

Write Rate vs Apply Rate

Let \(G\) be the rate at which the source generates replication work and \(A\) be the rate at which the replica applies it.

The replica can catch up when:

\[ A > G \]

Lag continues growing when:

\[ G > A \]

When generation stops or decreases, a simplified catch-up estimate is:

\[ CatchUpTime \approx \frac{ ReplicationBacklog }{ ApplyRate - GenerationRate } \]

This is a planning approximation. Transaction dependencies, parallelism, retries, network variation, and database implementation affect actual recovery.

Replica Read Workload

Read replicas are commonly used for reporting, search indexing, exports, and analytical queries. These workloads can compete with replication apply.

Replica resources
      |
      +-- Replication receiver
      +-- Replication apply
      +-- Application reads
      +-- Reporting queries
      +-- Backup operations
      +-- Maintenance tasks

Separate workload pools, query limits, dedicated analytical systems, or replica-specific routing can protect replication capacity.

Long-running Queries

A long-running replica query can hold resources or maintain an older database view. Depending on the database, this can interfere with cleanup, replication apply, or storage processing.

Review:

• Query duration
• Locks and transaction state
• Temporary storage
• CPU and I/O consumption
• Cancellation and statement timeouts
• Whether reporting should use another data path

Replication Slots and Log Retention

Some replication systems retain source log data until a subscriber confirms that the data is no longer required.

Source generates replication logs
      |
      v
Subscriber stops consuming
      |
      v
Required logs cannot be removed
      |
      v
Retained log volume grows
      |
      v
Source storage pressure increases

An inactive replication slot or subscriber can therefore create both lag and source-storage growth. Monitor subscriber state, retained log volume, and recovery progress.

Conceptual PostgreSQL Lag Inspection

SELECT
    application_name,
    client_addr,
    state,
    sent_lsn,
    write_lsn,
    flush_lsn,
    replay_lsn,
    write_lag,
    flush_lag,
    replay_lag
FROM pg_stat_replication;

The availability and meaning of fields depend on the PostgreSQL version, replication mode, and current activity. Interpret the status using the selected platform's official documentation.

Conceptual Position Difference

SELECT
    application_name,
    pg_wal_lsn_diff(
        sent_lsn,
        replay_lsn
    ) AS pending_replay_bytes
FROM pg_stat_replication;

A byte difference shows outstanding replication volume. It does not directly equal a fixed catch-up duration because apply throughput varies.

Lag Measured in Time vs Bytes

Read-routing Strategies

Learning-platform Examples

Do Not Use Stale Reads for Strict Invariants

Some decisions must use sufficiently current authoritative data.

Examples include:

• Checking whether an enrollment already exists
• Validating an active session after revocation
• Checking available inventory before reservation
• Confirming a payment state
• Enforcing a unique business reference
• Authorizing access after a permission removal

Reading a stale replica and then writing to the leader can produce a time-of-check to time-of-use problem. Enforce critical invariants through transactions, conditional writes, uniqueness constraints, locking, or another approved coordination mechanism at the authoritative write path.

Retry and Duplicate-operation Risk

Client submits enrollment
      |
      v
Leader commits enrollment
      |
      v
Client reads lagging replica
      |
      v
Enrollment appears missing
      |
      v
Client submits again

Protect state-changing operations with idempotency rather than relying on an immediate replica read to determine whether the original request succeeded.

Idempotent Request

POST /api/enrollments HTTP/1.1
Host: api.example.com
Authorization: Bearer access-token
Idempotency-Key: unique-operation-key
Content-Type: application/json

{
  "courseId": 42
}

Read Repair and Anti-Entropy

Leaderless and selected replicated systems can use repair mechanisms to bring stale replicas up to date.

Read Repair

Read several replicas
      |
      v
Detect an older version
      |
      v
Return selected current value
      |
      v
Update stale replica

Anti-Entropy

Background replica comparison
      |
      v
Identify missing or divergent ranges
      |
      v
Transfer required versions
      |
      v
Verify convergence

Read repair helps frequently accessed data. Background repair is needed for data that might not be read often enough to trigger repair.

Conceptual Freshness Policy

replicaReads:
  default:
    source: eligible-replica
    freshness: documented-eventual

  readYourWrites:
    minimumRequiredVersion: trusted-session-context

    fallback:
      - current-replica
      - leader
      - bounded-wait
      - controlled-failure

  monotonicReads:
    trackMinimumObservedVersion: true

  criticalOperations:
    source: authoritative-path
    allowStaleRead: false

  routing:
    maximumApprovedLag: workload-specific

  monitoring:
    replicaPositions: enabled
    lagByReplica: enabled
    staleReadEvents: enabled

This is conceptual configuration. Use the actual consistency, routing, and replication features provided by the selected database.

Security Impact

Replication lag affects security-sensitive state as well as business data.

Examples include:

• A revoked session remaining visible as active on a lagging replica
• A removed role remaining visible temporarily
• A disabled account appearing enabled
• A revoked API credential appearing valid
• An outdated access-control list permitting previous access

Security decisions should use a consistency level appropriate to the risk. Do not direct revocation-sensitive authorization checks to an arbitrarily stale replica.

Observability

Useful replication-lag metrics include:

• Lag time by replica
• Outstanding replication bytes
• Source and replica positions
• Replication receive rate
• Replication apply rate
• Pending replication operations
• Oldest unapplied transaction age
• Replication connection state
• Write, flush, and replay progress where available
• Stale-read count
• Read reroutes to leader
• Reads waiting for replica catch-up
• Read-repair and anti-entropy backlog
• Retained log or replication-slot volume
• Replica CPU, memory, I/O, and query load

Structured Lag Event

{
  "replica": "course-read-replica-2",
  "sourcePosition": "15800",
  "appliedPosition": "15720",
  "positionDifference": 80,
  "replicationState": "catching-up",
  "routingEligibility": "restricted"
}

Do not include credentials, connection secrets, or sensitive record values in replication-monitoring events.

Alert Conditions

Alert when:

• Lag exceeds the approved freshness objective
• Lag continues increasing rather than stabilizing
• Replication receive or apply stops
• A replica remains in catch-up state unexpectedly
• Retained replication logs threaten source storage capacity
• A replica repeatedly serves stale data beyond policy
• No replica satisfies freshness-sensitive reads
• Leader traffic increases because replicas remain ineligible
• Read repair or anti-entropy backlog grows
• Failover candidates remain substantially behind
• Replica CPU, I/O, or storage becomes saturated
• Lag increases after a schema or deployment change

Troubleshooting Workflow

1. Identify the replication topology and affected replicas.
2. Determine whether lag is measured in time, bytes, positions, or transactions.
3. Compare source generation rate with replica receive and apply rates.
4. Check replication connection and subscriber state.
5. Check network latency, interruption, congestion, and bandwidth.
6. Check replica CPU, memory, disk latency, and storage throughput.
7. Check long-running queries and lock contention.
8. Check large transactions and bulk changes.
9. Check recent schema, index, deployment, and configuration changes.
10. Check replication slots and retained logs.
11. Reduce or isolate nonessential replica workloads where safe.
12. Confirm that freshness-sensitive reads avoid the lagging replica.
13. Estimate whether the replica can catch up at its current apply rate.
14. Repair or rebuild the replica through the approved database procedure if necessary.

Common Replication-lag Mistakes

Recommended Test Cases

Replication-lag Best Practices

Practice Exercise

Design replication-lag handling for your online learning platform.

Requirements

1. Identify which data is read from replicas.
2. Define acceptable freshness for course catalog data.
3. Provide read-your-writes after a learner updates progress.
4. Prevent progress from moving backward across replica reads.
5. Preserve ordering between discussion questions and replies.
6. Keep enrollment validation on an authoritative path.
7. Protect session and permission revocation from stale reads.
8. Capture the write position required by a subsequent read.
9. Route reads only to replicas meeting the required position.
10. Define a bounded fallback when no replica has caught up.
11. Add idempotency to enrollment and progress-submission requests.
12. Monitor lag time, bytes, positions, and apply state.
13. Test a large bulk course update.
14. Test a disconnected replica.
15. Test leader failover while followers are behind.

Freshness-policy Template

Frequently Asked Questions