Introduction

Producers and consumers communicate across networks and independent processes. Any part of this communication can fail.

Producer
    |
    v
Messaging system
    |
    v
Consumer
    |
    v
Database or external service

Possible failures include:

• The producer sends a message but does not receive confirmation.
• The broker stores a message but the acknowledgment is lost.
• The consumer receives a message and then crashes.
• The business operation succeeds but the consumer acknowledgment fails.
• The consumer commits its progress before completing the operation.
• A network timeout leaves the caller uncertain about the result.

Delivery semantics define how a messaging system and its applications handle these uncertain situations.

The three commonly discussed delivery semantics are:

• At-most-once
• At-least-once
• Exactly-once

Prerequisites

What Are Delivery Semantics?

Delivery semantics describe the observable behaviour of a message when producers, brokers, consumers, networks, or downstream systems fail.

A complete messaging path contains several separate boundaries:

1. Producer creates message

2. Producer sends message to broker

3. Broker stores message

4. Broker delivers message to consumer

5. Consumer processes message

6. Consumer updates business state

7. Consumer records message progress

8. Broker considers processing complete

A system can provide a strong guarantee at one boundary and a weaker guarantee at another.

Delivery Semantics Comparison

At-Most-Once Delivery

At-most-once delivery means a message is processed zero or one time. The system avoids redelivery, but a failure can cause the message or business operation to be lost.

Consumer receives message
      |
      v
Consumer records progress
or acknowledges immediately
      |
      v
Consumer starts processing
      |
      v
Consumer crashes
      |
      v
Message is not delivered again
      |
      v
Business operation is lost.

Advantages

• No duplicate processing caused by messaging retries
• Simple consumer behaviour
• Can reduce acknowledgment and retry overhead
• Suitable when occasional loss is acceptable

Limitations

• A message can be lost during failure
• A business operation can be skipped permanently
• Not appropriate for critical financial or transactional work
• Recovery can require another authoritative source

Possible Use Cases

• Noncritical telemetry samples
• Approximate metrics
• High-frequency signals where later values supersede earlier values
• Optional diagnostic data

At-Least-Once Delivery

At-least-once delivery retries a message when successful delivery or processing cannot be confirmed.

Consumer receives message
      |
      v
Consumer performs business update
      |
      v
Business update commits
      |
      v
Consumer crashes before
acknowledgment or offset commit
      |
      v
Message is delivered again.

The message is less likely to be silently lost, but duplicate processing can occur.

Advantages

• Retries protect against transient failures
• Required work is less likely to disappear silently
• Suitable for many production workflows
• Works well with idempotent consumers

Limitations

• The same logical message can be processed more than once
• External side effects can be repeated
• Consumers require idempotency or deduplication
• Poison messages require bounded retry handling

Common Use Cases

• Database projection updates
• Order-processing workflows
• Search-index updates
• Report-generation tasks
• Webhook delivery
• Change Data Capture pipelines

Exactly-Once Semantics

Exactly-once semantics means that a record produces one result inside a clearly defined processing boundary, even when failures and retries occur.

The phrase must always include the boundary.

Possible exactly-once boundary:

Kafka input topic
      |
      v
Kafka transaction
      |
      +-- Consume input records
      +-- Produce output records
      +-- Commit consumed offsets
      |
      v
Kafka output topic

Kafka can coordinate transactional record production and consumed-offset progress for supported Kafka processing flows. This does not automatically make an unrelated database, payment provider, email service, or HTTP API part of the same transaction.

Kafka transaction:

Exactly-once processing
inside Kafka boundary


External payment API:

Separate failure boundary


External database:

Separate transaction boundary

Effectively-Once Business Outcome

Effectively-once processing accepts that delivery can happen more than once while ensuring that repeated deliveries do not create repeated business effects.

Message delivered twice
      |
      v
Consumer checks stable message ID
      |
      +-- First delivery:
      |      apply business operation
      |      record message ID
      |
      +-- Repeated delivery:
             detect existing message ID
             return previous outcome

This pattern is commonly more practical when a consumer writes to a database or calls an external service that is outside the broker's transaction.

Exactly-Once vs Effectively-Once

Delivery Has Multiple Boundaries

A messaging workflow should be analyzed one boundary at a time.

Producer-side Uncertainty

Producer sends message
      |
      v
Broker stores message
      |
      v
Network fails before
acknowledgment reaches producer
      |
      v
Producer cannot know whether
the message was stored.

The producer has two broad choices:

• Do not retry, accepting possible message loss
• Retry, accepting possible duplication unless the broker or producer prevents it

Consumer-side Uncertainty

Consumer receives event
      |
      v
Consumer updates database
      |
      v
Database commits
      |
      v
Consumer crashes before
acknowledgment or offset commit
      |
      v
Event is delivered again.

At-least-once systems need consumer-side duplicate protection because this failure window cannot be eliminated by retry alone.

Kafka Delivery Semantics

Kafka delivery behaviour depends on producer acknowledgments and retries, consumer offset timing, idempotent production, and transactional processing.

Kafka At-Most-Once Consumer

Poll Kafka records
      |
      v
Commit offsets
      |
      v
Process records
      |
      v
Failure after commit
      |
      v
Records are skipped after restart.

Kafka At-Least-Once Consumer

Poll Kafka records
      |
      v
Process records
      |
      v
Commit business state
      |
      v
Commit offsets
      |
      v
Failure before offset commit
can repeat records.

Kafka Transactional Processing

Consume input records
      |
      v
Begin Kafka transaction
      |
      +-- Produce output records
      +-- Include consumed offsets
      |
      v
Commit Kafka transaction

This model can provide exactly-once processing for supported Kafka-to-Kafka pipelines. An external database or HTTP service remains a separate boundary unless another coordination mechanism is used.

Kafka Consumer Configurations

kafkaConsumer:
  groupId: learning-progress-projection

  offsetManagement:
    automaticCommit: false
    commitAfterDurableProcessing: true

  processing:
    idempotencyRequired: true
    boundedBatchSize: true

  retries:
    boundedAttempts: true
    backoff: exponential-with-jitter
    deadLetterPath: approved-failure-topic

  observability:
    consumerLag: enabled
    commitFailures: enabled
    duplicateDetections: enabled
    processingFailures: enabled

RabbitMQ Delivery Semantics

RabbitMQ separates producer-to-broker confirmation from consumer-to-broker processing confirmation.

Publisher Confirm

Publisher sends message
      |
      v
RabbitMQ accepts publication
according to configured policy
      |
      v
RabbitMQ sends publisher confirm

A publisher confirm does not prove that any consumer completed the business operation.

Consumer Acknowledgment

RabbitMQ delivers message
      |
      v
Consumer processes message
      |
      +-- Success:
      |      acknowledge
      |
      +-- Temporary failure:
      |      reject or negatively acknowledge
      |      using approved retry behaviour
      |
      +-- Permanent failure:
             route to controlled
             dead-letter handling

Automatic Acknowledgment Risk

Message delivered
      |
      v
Broker treats message
as acknowledged
      |
      v
Consumer begins processing
      |
      v
Consumer crashes
      |
      v
Work can be lost.

Manual Acknowledgment and Redelivery

Consumer processes message
      |
      v
Business transaction commits
      |
      v
Connection fails before acknowledgment
      |
      v
RabbitMQ can redeliver message.

Manual acknowledgment after durable processing commonly produces at-least-once behaviour and therefore requires idempotency.

Amazon SQS Delivery Semantics

Amazon SQS uses receive, visibility timeout, and delete operations to control message processing.

SQS Standard Queue

Standard queues use at-least-once delivery. A message can be received more than once, so consumers must be idempotent.

Consumer receives message
      |
      v
Message becomes temporarily invisible
      |
      v
Consumer processes message
      |
      +-- Success:
      |      delete message
      |
      +-- Failure or no delete:
             visibility expires
             message becomes available again

SQS FIFO Queue

FIFO queues provide FIFO processing and duplicate-suppression features within the documented SQS FIFO model. Message groups provide separate ordered sequences.

Message Group:

learner-1042


Messages:

RegisterAccount
EnrollInCourse
CompleteCourse

Application-side idempotency remains valuable because a business operation can succeed while the message deletion or external response remains uncertain.

Platform Comparison

Idempotency with a Database

A common approach records the message ID and the business update in one database transaction.

Begin database transaction
      |
      v
Check processed-message record
      |
      +-- Already processed:
      |      return existing result
      |
      +-- New message:
             apply business update
             insert processed-message record
      |
      v
Commit transaction
      |
      v
Acknowledge, commit offset,
or delete queue message

Deduplication Table

CREATE TABLE processed_messages
(
    consumer_name VARCHAR(150) NOT NULL,
    message_id VARCHAR(150) NOT NULL,
    processed_at TIMESTAMP NOT NULL,

    PRIMARY KEY
    (
        consumer_name,
        message_id
    )
);

Version-aware Business Table

CREATE TABLE learner_progress
(
    tenant_id BIGINT NOT NULL,
    learner_id BIGINT NOT NULL,
    course_id BIGINT NOT NULL,
    progress_percentage DECIMAL(5, 2) NOT NULL,
    source_version BIGINT NOT NULL,

    PRIMARY KEY
    (
        tenant_id,
        learner_id,
        course_id
    )
);

Deduplication prevents the same message from being applied twice. Version checks can additionally prevent an older replay from overwriting newer state.

Conceptual Idempotent Consumer

public void process(Message message) {
    database.transaction(() -> {
        boolean alreadyProcessed =
            processedMessages.exists(
                "progress-projection",
                message.getMessageId()
            );

        if (alreadyProcessed) {
            return;
        }

        progressRepository.applyIfNewer(
            message.getTenantId(),
            message.getLearnerId(),
            message.getCourseId(),
            message.getProgressPercentage(),
            message.getSourceVersion()
        );

        processedMessages.record(
            "progress-projection",
            message.getMessageId()
        );
    });

    acknowledgeMessagingProgress(message);
}

This is an educational example. Production processing requires safe transaction boundaries, schema validation, exception classification, authorization, retries, shutdown handling, and platform-specific message progress logic.

External Side Effects

Calling an external system introduces another uncertain boundary.

Consumer calls payment API
      |
      v
Payment provider completes charge
      |
      v
Network response is lost
      |
      v
Consumer does not know whether
the charge succeeded.

Retrying without protection can create a duplicate charge.

Possible controls include:

• An idempotency key accepted by the external service
• A stable business operation identifier
• A status-query API before retry
• A local state machine for uncertain outcomes
• A reconciliation process

Email and Notification Side Effects

Consumer sends email
      |
      v
Email service accepts request
      |
      v
Consumer crashes before
recording success
      |
      v
Message is retried
      |
      v
Recipient can receive
the email again.

Use a stable notification identifier and a provider or application-side deduplication policy when duplicate notifications are unacceptable.

Transactional Outbox

A transactional outbox coordinates a local business update with recording the message that must be published.

Begin local transaction
      |
      +-- Update business data
      +-- Insert outbox message
      |
      v
Commit once
      |
      v
Outbox publisher sends message
      |
      v
Publication can be retried safely

Outbox Table

CREATE TABLE message_outbox
(
    message_id VARCHAR(150) PRIMARY KEY,
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id VARCHAR(150) NOT NULL,
    message_type VARCHAR(150) NOT NULL,
    payload TEXT NOT NULL,
    created_at TIMESTAMP NOT NULL,
    published_at TIMESTAMP NULL
);

The outbox prevents a local transaction from committing without leaving a durable publication record. The publisher and consumer still require duplicate-safe behaviour.

Retries

Retries are required for temporary faults but should remain bounded and classified.

Processing failure
      |
      v
Classify failure
      |
      +-- Temporary:
      |      retry with backoff
      |
      +-- Permanent:
      |      dead-letter or reject
      |
      +-- Unknown:
             preserve evidence
             use controlled handling

Use:

• Maximum attempt limits
• Exponential backoff
• Randomized jitter
• Dead-letter destinations
• Operator-visible failure reasons
• Controlled redrive

Poison Messages

A poison message fails consistently and cannot succeed through immediate retry.

Invalid message
      |
      v
Consumer fails
      |
      v
Immediate retry
      |
      v
Consumer fails again
      |
      v
Useful processing is delayed.

Infinite retry is not a reliability strategy. Move terminal failures to a controlled failure path while preserving the data needed for diagnosis.

Ordering and Delivery Semantics

Ordering and delivery count are separate guarantees.

Deduplication Is Not Ordering

Event Version 5 arrives

Event Version 4 arrives later


Both event IDs are unique.


Deduplication alone:

Does not detect that Version 4
is older than Version 5.

Use version checks, sequence numbers, logical timestamps, or conditional updates when older events must not overwrite newer state.

Exactly-Once Does Not Mean Error-Free

Exactly-once processing does not guarantee that:

• The business logic is correct
• The event schema contains correct data
• The partition key is correct
• The external API behaves idempotently
• A user receives one notification
• A database outside the transaction receives one update
• An operator never replays the event manually

Exactly-once semantics solve a defined processing problem, not every source of application duplication or incorrectness.

Learning-platform Examples

Conceptual Delivery Policy

deliveryPolicy:
  semantic: at-least-once

  producer:
    stableMessageId: required
    retry:
      enabled: true
      maximumAttempts: approved-limit
      backoff: exponential-with-jitter

  consumer:
    progressAfterDurableProcessing: true
    idempotency: required
    schemaValidation: required

  businessState:
    deduplication:
      storage: transactional-database
      uniqueConstraint: required

    versionCheck:
      enabled: true

  externalEffects:
    idempotencyKey: required-when-supported
    reconciliation: required-for-uncertain-results

  poisonMessages:
    boundedRetries: true
    deadLetterPath: approved-destination

  observability:
    duplicateDetection: enabled
    retryCount: enabled
    deadLetterCount: enabled
    uncertainOutcomes: enabled

This configuration is conceptual. Exact acknowledgments, transactions, visibility, offset, and deduplication behaviour must be verified against the selected broker and client version.

Observability

Useful delivery-semantics metrics include:

• Messages published
• Producer acknowledgment failures
• Producer retry count
• Consumer delivery count
• Duplicate-message detections
• Consumer acknowledgment failures
• Offset-commit failures
• Visibility-timeout expirations
• Messages processed successfully
• Business-operation failures
• Retry attempts by failure category
• Dead-letter messages
• Uncertain external outcomes
• Reconciliation mismatches
• Oldest unprocessed-message age

Structured Delivery Event

{
  "messageId": "stable-message-id",
  "consumer": "certificate-generator",
  "deliveryAttempt": 2,
  "processingResult": "duplicate-absorbed",
  "businessOutcome": "already-completed",
  "deliverySemantic": "at-least-once"
}

Avoid including reusable credentials, complete private payloads, payment details, or sensitive personal data in routine delivery logs.

Alert Conditions

Alert when:

• Producer retries increase unexpectedly
• Publisher acknowledgments or confirms repeatedly fail
• Consumer redelivery increases
• Duplicate-message detections increase
• Offset commits or queue acknowledgments repeatedly fail
• SQS messages repeatedly return after visibility timeout
• Dead-letter volume grows
• Poison messages block useful processing
• Business outcomes remain uncertain after external timeouts
• Reconciliation detects missing or duplicated operations
• Oldest-message age exceeds the processing objective
• Retry traffic threatens downstream-system capacity

Troubleshooting Workflow

1. Identify the message ID and business operation ID.
2. Identify the intended delivery semantic.
3. Define the producer, broker, consumer, and external-system boundaries.
4. Check whether the broker accepted the publication.
5. Check producer acknowledgment or confirm results.
6. Check the consumer's delivery attempts.
7. Check acknowledgment, offset, or deletion timing.
8. Check whether the business database transaction committed.
9. Check the processed-message or idempotency record.
10. Check version or sequence information.
11. Check retries and dead-letter handling.
12. Check external-service status using the stable operation ID.
13. Reconcile missing or duplicate effects through the approved procedure.

Common Delivery-semantics Mistakes

Recommended Test Cases

Delivery-semantics Best Practices

Practice Exercise

Design delivery semantics for asynchronous workflows in your online learning platform.

Requirements

1. Classify click telemetry as critical or loss-tolerant.
2. Define delivery semantics for learner-progress events.
3. Create a stable message ID for every event.
4. Store progress updates and processed-message IDs transactionally.
5. Prevent older progress events from overwriting newer progress.
6. Define SQS message deletion after durable report creation.
7. Define RabbitMQ acknowledgment after certificate generation.
8. Define Kafka offset commit after projection persistence.
9. Use Kafka transactions for one Kafka-to-Kafka transformation.
10. Document the exact boundary of the Kafka guarantee.
11. Add an idempotency key to an external notification request.
12. Use an outbox for enrollment-event publication.
13. Create bounded retry and dead-letter policies.
14. Simulate failure before processing.
15. Simulate failure after the business commit.
16. Verify that duplicate delivery creates one business outcome.

Design Template

Frequently Asked Questions