Prerequisites

Before studying batch recomputation, you should have a basic understanding of the following concepts:

What Is Batch Recomputation?

Batch recomputation is the process of rebuilding a derived dataset by reading a bounded collection of source data and recalculating some or all of the output. The computation usually runs on a schedule, after an important data change, or when an operator initiates a backfill or repair.

A derived dataset is information calculated from other data. It may include a search index, product ranking, recommendation list, daily report, machine-learning feature table, autocomplete dictionary, or materialized aggregate.

The function \(f\) represents the transformation logic. During a full recomputation, the system applies this logic to the complete relevant source dataset and produces a new version of the derived state.

Authoritative Data and Derived Data

A well-designed recomputation pipeline clearly separates authoritative data from derived data.

Why Batch Recomputation Matters in Search and Ranking

Search and ranking systems frequently depend on information collected from many sources. Examples include document content, product metadata, click activity, popularity, freshness, quality, availability, and business rules.

Updating every output immediately after every source change may be too expensive or unnecessarily complex. A batch job can periodically combine the latest source data and build a consistent output snapshot.

How a Batch Recomputation Pipeline Works

A production batch-recomputation workflow usually contains multiple controlled stages. Each stage should be observable, restartable, and protected from incomplete output.

Full Recomputation

Full recomputation processes the entire relevant source dataset and rebuilds the complete output. It is conceptually simple because the final result depends on the current source data rather than a complicated history of updates.

Incremental Recomputation

Incremental recomputation processes only new or changed input and applies the resulting differences to existing derived state. It generally reduces work, but it introduces additional state and correctness challenges.

Here, \(O_t\) is the new output, \(O_{t-1}\) is the previous output, and \(\Delta(I_t)\) represents the changes calculated from new, modified, or deleted input records.

The formula uses addition conceptually. In practice, applying a delta may involve inserts, updates, deletions, score adjustments, or replacement of affected partitions.

Choosing Between Full and Incremental Processing

The decision should be based on freshness requirements, dataset size, update frequency, processing cost, operational complexity, and the cost of an incorrect result.

Example: Recomputing Product Ranking

Consider a product-search system that recalculates a ranking score from textual relevance, popularity, freshness, and product quality.

In this example:

• \(R\) represents normalized textual relevance.
• \(P\) represents normalized popularity.
• \(F\) represents normalized freshness.
• \(Q\) represents normalized product quality.

The exact weights are business decisions and should be evaluated using relevance metrics, experiments, and user feedback rather than copied blindly.

CREATE TABLE product_ranking_build (
    build_id         VARCHAR(50) NOT NULL,
    product_id       BIGINT NOT NULL,
    relevance_score DECIMAL(10, 6) NOT NULL,
    popularity_score DECIMAL(10, 6) NOT NULL,
    freshness_score DECIMAL(10, 6) NOT NULL,
    quality_score   DECIMAL(10, 6) NOT NULL,
    final_score     DECIMAL(10, 6) NOT NULL,
    computed_at     TIMESTAMP NOT NULL,
    PRIMARY KEY (build_id, product_id)
);

INSERT INTO product_ranking_build (
    build_id,
    product_id,
    relevance_score,
    popularity_score,
    freshness_score,
    quality_score,
    final_score,
    computed_at
)
SELECT
    'ranking-2026-09-24',
    p.product_id,
    p.relevance_score,
    p.popularity_score,
    p.freshness_score,
    p.quality_score,
    (0.45 * p.relevance_score) +
    (0.25 * p.popularity_score) +
    (0.20 * p.freshness_score) +
    (0.10 * p.quality_score),
    CURRENT_TIMESTAMP
FROM product_features p
WHERE p.is_active = TRUE;

The build identifier keeps one recomputation separate from another. The new result can be validated before the serving layer starts using it.

Example Pipeline Pseudocode

async function runBatchRecomputation(cutoffTime) {
    const buildId = createBuildId(cutoffTime);

    await registerBuild({
        buildId: buildId,
        cutoffTime: cutoffTime,
        status: "RUNNING"
    });

    try {
        const input = await readSourceSnapshot(cutoffTime);

        await validateSourceData(input);

        const partitions = partitionInput(input);

        await processInParallel(partitions, async function (partition) {
            const output = recomputeDerivedRecords(partition);

            await writeToStaging(buildId, output);
        });

        const validation = await validateStagingOutput(buildId);

        if (!validation.isValid) {
            throw new Error("Output validation failed");
        }

        await publishBuildAtomically(buildId);

        await markBuildComplete(buildId);
    } catch (error) {
        await markBuildFailed(buildId, error.message);
        throw error;
    }
}

This workflow writes every result into staging storage and publishes only after validation succeeds. A retry can reuse or safely replace output associated with the same build identifier.

Idempotency and Duplicate Protection

Batch jobs must be safe to retry. A worker may finish writing its output but fail before reporting success. Without idempotency, a retry could create duplicate records or apply the same aggregate more than once.

In practical terms, processing the same input for the same build multiple times should not produce additional or inconsistent output.

Partitioning the Work

Large datasets are divided into partitions so multiple workers can process them in parallel. Common partitioning strategies include:

Balanced partitions improve parallelism. Poor partitioning can create stragglers, where most workers finish early while one worker continues processing a disproportionately large or expensive partition.

The slowest partition often determines the completion time of the whole stage. Partition-size distribution is therefore as important as the average amount of work.

Safe Publication with Versioned Outputs

Consumers should not read the output while it is only partially built. A safer design creates an immutable version and changes a small pointer only after validation succeeds.

{
    "activeBuild": "ranking-2026-09-24",
    "previousBuild": "ranking-2026-09-23",
    "sourceCutoff": "2026-09-24T00:00:00Z",
    "status": "PUBLISHED"
}

The active-build pointer may be stored in a database, metadata service, search alias, configuration store, or catalog. The publication operation should be atomic from the consumer's perspective.

Checkpoints, Manifests, and Resumability

A large recomputation should not necessarily restart from the beginning after a single worker fails. The pipeline can record progress at stable boundaries.

A manifest describes the expected input, output partitions, code version, source cutoff, and completion state of a particular build.

{
    "buildId": "search-index-2026-09-24",
    "sourceCutoff": "2026-09-24T00:00:00Z",
    "transformationVersion": "ranking-v4",
    "expectedPartitions": 4,
    "completedPartitions": [
        "partition-00",
        "partition-01",
        "partition-03"
    ],
    "pendingPartitions": [
        "partition-02"
    ],
    "status": "RUNNING"
}

After a failure, the orchestrator can retry the missing partition instead of processing every successful partition again. The resumed job must still verify that the input snapshot and transformation version have not changed.

Freshness and Recomputation Frequency

Batch output represents source data only up to a particular cutoff. The difference between the current time and the newest source data reflected in the output is commonly described as data staleness.

Increasing recomputation frequency can improve freshness, but it also increases compute usage, operational load, and the chance that one run overlaps with the next.

Common Failure Scenarios

Validation Before Publication

A job completing successfully does not prove that its output is correct. Technical and business-level validations should run before publication.

Monitoring and Observability

Monitoring should explain whether the job started, what it is processing, where time is being spent, whether output is valid, and whether the new version was published.

Capacity and Cost Estimation

A basic estimate can help determine whether a full recomputation can finish within the available processing window.

In this simplified model:

• \(D\) is the total amount of input data.
• \(N\) is the number of workers.
• \(P\) is the effective throughput of one worker.
• \(E\) is the parallel-efficiency factor, where \(0 < E \leq 1\).

Real systems also include startup time, data shuffling, skew, throttling, retries, output commits, and validation. Therefore, this formula should be treated as an initial estimate rather than a guaranteed runtime.

Optimization Techniques

Example: Idempotent Partition Replacement

One safe pattern is to compute a complete partition in staging and replace the corresponding destination partition only after the staged data passes validation.

BEGIN;

DELETE FROM daily_product_metrics
WHERE metric_date = '2026-09-23';

INSERT INTO daily_product_metrics (
    metric_date,
    product_id,
    views,
    clicks,
    purchases
)
SELECT
    metric_date,
    product_id,
    views,
    clicks,
    purchases
FROM daily_product_metrics_staging
WHERE build_id = 'metrics-2026-09-24'
  AND metric_date = '2026-09-23';

COMMIT;

Replacing the complete destination partition avoids adding the same aggregation twice. For very large partitions, the platform may offer more efficient partition-exchange or versioned-table mechanisms.

Example System Design

Consider a product-search platform that receives product records, catalog updates, inventory changes, click events, and purchase events. It needs to periodically refresh both its search index and ranking features.

Proposed Flow

1. Business services write authoritative product and transaction data.
2. Raw records and events are copied into durable storage.
3. The orchestrator creates a build identifier and source cutoff.
4. Distributed workers clean records and calculate ranking features.
5. Indexing workers create a new version of the search index.
6. Automated checks validate document counts, schema, permissions, and known search queries.
7. The search alias is atomically moved to the new index.
8. The previous index remains available for rollback.

System Design Interview Discussion

During a system design interview, avoid saying only that a cron job will rebuild the index. Explain the correctness and operational decisions behind the job.

Best-Practice Checklist

Knowledge Check

Summary

Batch recomputation rebuilds derived state from a bounded set of authoritative data. It is commonly used for search indexes, ranking features, recommendations, analytics, autocomplete, and materialized aggregates.

Full recomputation offers a simpler correctness model and can repair accumulated errors, but it becomes expensive as data grows. Incremental recomputation reduces repeated work and can improve freshness, but it requires careful handling of state, deletions, retries, late data, and duplicates.

A reliable design uses stable input boundaries, versioned transformation logic, balanced partitions, idempotent workers, checkpoints, staging output, comprehensive validation, atomic publication, monitoring, and rollback support.