Introduction

Distributed services often need to communicate without waiting for every downstream operation to finish inside the original request.

Client submits an operation
      |
      v
Application validates request
      |
      v
Message is published
      |
      v
Application responds
      |
      v
Background consumer processes message

Asynchronous messaging separates the producer that creates work or events from the consumer that processes them.

Two common messaging models are:

• Message queues
• Append-only event logs

Both models carry messages between producers and consumers, but they answer an important question differently:

What should happen to a message
after one consumer processes it?

In the traditional queue model, consumers compete for work. One consumer processes a queued message, acknowledges it, and the broker can remove it according to its policy.

In the log model, reading does not remove the event. The event remains in the retained log, and each consumer group tracks its own reading position.

Prerequisites

What Is a Message Queue?

A message queue stores messages that represent work waiting for an eligible consumer.

Producer
    |
    v
+-----------------------+
|         Queue         |
| Task 1                |
| Task 2                |
| Task 3                |
+-----------------------+
    |
    +-- Consumer A
    +-- Consumer B
    +-- Consumer C

Consumers connected to the same queue normally compete for messages. A particular message is assigned to one consumer at a time according to the broker's delivery policy.

Typical queue workloads include:

• Sending an email
• Generating a report
• Resizing an uploaded image
• Processing a video
• Executing an integration request
• Running a background database operation
• Dispatching a notification

What Is an Event Log?

An event log is an append-only sequence of records. New events are appended to the end of a partition, and retained events remain available according to a time-based, size-based, or compaction policy.

Partition 0:

Offset 0 -> CourseCreated
Offset 1 -> CoursePublished
Offset 2 -> LearnerEnrolled
Offset 3 -> ProgressUpdated
Offset 4 -> CertificateIssued

A consumer reads records in sequence and tracks its position, commonly called an offset.

Consumer Group A:

Current offset = 5


Consumer Group B:

Current offset = 2


Consumer Group C:

Current offset = 0

The consumer groups can read the same event history independently and at different speeds.

Core Difference

Queue Processing Model

Queue contains:

Task A
Task B
Task C
Task D


Consumers:

Worker 1 receives Task A

Worker 2 receives Task B

Worker 3 receives Task C


Task D remains waiting.

Adding workers increases processing parallelism until another resource becomes the bottleneck.

Each task should normally be handled by one successful worker. If the worker fails before acknowledgment, the broker can make the task available for another delivery according to its policy.

Log Consumption Model

Event log:

Event A
Event B
Event C
Event D


Consumer Group: Notifications

Reads A, B, C, D


Consumer Group: Analytics

Reads A, B, C, D


Consumer Group: Search Index

Reads A, B, C, D

The groups read the same retained records without requiring the producer to write a separate copy for each group.

Competing Consumers

A queue commonly distributes different messages among several workers.

                  Queue
               /    |    \
              /     |     \
             v      v      v

         Worker A Worker B Worker C


One message is completed
by one successful worker.

This pattern is useful when workers provide equivalent processing capability and each task needs one successful execution.

Consumer Groups

A log consumer group combines queue-like competition with log-based retention.

Topic with four partitions:

Partition 0
Partition 1
Partition 2
Partition 3


Consumer Group A:

Consumer A1 -> Partition 0
Consumer A2 -> Partition 1
Consumer A3 -> Partition 2
Consumer A4 -> Partition 3

Consumers inside one group divide partition ownership. Another group can independently consume all four partitions for a different purpose.

Queue Acknowledgments

An acknowledgment tells the queue broker that a delivered message was processed successfully.

Broker delivers message
      |
      v
Consumer processes message
      |
      +-- Success:
      |      acknowledge
      |      broker completes message
      |
      +-- Failure:
             reject, retry,
             or allow redelivery

Acknowledging before the business operation commits can lose work if the consumer fails after acknowledgment.

Acknowledging after the operation commits can produce redelivery if the acknowledgment is lost.

Consumers should therefore be idempotent.

Log Offsets

An offset identifies a record's position within a log partition.

Partition 2:

Offset 100 -> Event A

Offset 101 -> Event B

Offset 102 -> Event C


Consumer committed offset:

101

The consumer uses its committed position to determine where processing should continue after restart or reassignment.

Advancing the offset before processing completes can skip work after a failure. Advancing it after processing completes can cause the event to be processed again after an uncertain failure.

Replay

Replay means moving a log consumer's position backward and processing retained events again.

Current consumer offset:

10,000


Replay starting offset:

8,000


Consumer processes:

8,000
8,001
8,002
...
10,000

Replay can support:

• Rebuilding a search index
• Recreating a materialized view
• Reprocessing events after correcting a consumer defect
• Producing a new analytical projection
• Onboarding a new consumer from retained history

Replay is safe only when consumers can handle repeated events and external side effects are protected.

Retention

Logs retain records according to an explicit policy rather than deleting each event when one consumer reads it.

Retention can be determined by:

• Event age
• Total retained size
• Per-partition size
• Key-based compaction
• Compliance and business requirements

Replay is possible only while the required event range remains available.

Log Compaction

Compaction retains a selected latest record for each key rather than every historical record indefinitely.

Original log:

course:42 -> Draft
course:42 -> Reviewed
course:42 -> Published


Compacted representation:

course:42 -> Published

Compaction can support state reconstruction, but it is not equivalent to a complete audit history because older values can be removed according to the compaction policy.

Queue Deletion vs Log Retention

Ordering in Queues

A queue can provide first-in-first-out behaviour under a specific scope, but concurrency, retries, priorities, and redelivery can affect the order in which processing completes.

Queue delivery:

Task 1 -> Worker A

Task 2 -> Worker B


Processing duration:

Task 1 takes 10 seconds

Task 2 takes 1 second


Completion order:

Task 2
Task 1

Delivery order and completion order are different concepts.

Ordering in Logs

A log normally provides ordering within a partition.

Partition 0:

Offset 1
Offset 2
Offset 3
Offset 4

Events in different partitions do not automatically share one global order.

Partition A:

A1
A2
A3


Partition B:

B1
B2
B3


No automatic total order exists
between A2 and B2.

Events requiring ordered processing should use a partition key that places related events in the same partition.

Partition Keys

A log producer commonly supplies a key that determines the destination partition.

Event key:

learner-1042


All related progress events:

ProgressStarted
LessonCompleted
QuizCompleted
CourseCompleted


Route to the same partition.

This can preserve order for one learner while allowing events for other learners to be processed in parallel.

Hot Partitions

A partition key with highly uneven traffic can create a hot partition.

Normal course keys:

1,000 events per minute


One large course key:

500,000 events per minute


Result:

The partition owning that key
becomes overloaded.

Adding consumers cannot increase parallel processing for one partition beyond the broker and consumer model's partition-level limit.

Possible controls include:

• Selecting a higher-cardinality partition key
• Splitting safely independent workloads into subkeys
• Using controlled bucketing
• Adding partitions before capacity is exhausted
• Applying producer rate limits
• Separating heavy workloads into another topic

Queue Parallelism

Queue workers commonly draw messages from one waiting-work collection.

Queue backlog:

100,000 tasks


Workers:

10 workers
      |
      v
Workers compete for
available tasks.

Additional workers can increase throughput until the queue, database, storage, network, or downstream service reaches its safe concurrency limit.

Log Parallelism

Partition count commonly determines the maximum active partition-processing parallelism within one consumer group.

Topic:

4 partitions


Consumer group:

2 consumers
    -> Each consumer handles
       multiple partitions


Consumer group:

4 consumers
    -> One active consumer
       per partition


Consumer group:

6 consumers
    -> Some consumers have
       no active partition

Increasing consumers beyond the relevant partition count does not create additional active partition ownership in that group.

Consumer-group Rebalancing

When a log consumer joins, leaves, or fails, partition ownership can be reassigned among the remaining consumers.

Consumer Group:

Consumer A -> Partitions 0 and 1

Consumer B -> Partitions 2 and 3


Consumer B fails
      |
      v
Group rebalances
      |
      v
Consumer A or a replacement
receives Partitions 2 and 3

Consumers must stop using revoked partitions, resume from committed offsets, and handle repeated processing safely around reassignment.

Delivery Guarantees

Idempotent Consumers

An idempotent consumer safely recognizes or absorbs repeated delivery of the same logical operation.

{
  "eventId": "unique-event-id",
  "eventType": "LearnerEnrolled",
  "tenantId": 17,
  "learnerId": 1042,
  "courseId": 42
}

The consumer can record the event identifier in the same transactional boundary as the resulting business update.

Conceptual Deduplication Table

CREATE TABLE processed_messages
(
    consumer_name VARCHAR(100) NOT NULL,
    message_id VARCHAR(150) NOT NULL,
    processed_at TIMESTAMP NOT NULL,

    PRIMARY KEY
    (
        consumer_name,
        message_id
    )
);

Dead-letter Handling

A message that repeatedly fails should not block useful work indefinitely.

Message delivery
      |
      v
Processing fails
      |
      v
Bounded retry policy
      |
      +-- Later success:
      |      complete message
      |
      +-- Retry limit reached:
             move to dead-letter
             handling path

A dead-letter workflow should preserve:

• Message identity
• Failure reason
• Attempt count
• Original destination
• Relevant correlation context
• Safe replay or disposal procedure

Avoid including secrets or unnecessary private data in dead-letter metadata.

Poison Messages

A poison message consistently fails because of invalid data, unsupported format, missing dependency, or a consumer defect.

Poison message
      |
      v
Consumer fails
      |
      v
Immediate retry
      |
      v
Consumer fails again
      |
      v
Partition or queue progress
is repeatedly disrupted

Use bounded retries, backoff, dead-letter handling, and operator-visible diagnostics.

Backpressure

Backpressure communicates that consumers cannot safely keep pace with producers.

Producer rate:

10,000 messages per second


Consumer capacity:

4,000 messages per second


Difference:

6,000 messages per second
accumulate in backlog.

A simplified backlog-growth estimate is:

\[ BacklogGrowthRate = ProducerRate - ConsumerCompletionRate \]

When producer rate remains greater than completion rate, backlog and processing delay continue to grow.

Queue Backlog

Useful queue pressure signals include:

• Visible message count
• Oldest-message age
• In-flight message count
• Consumer completion rate
• Retry rate
• Dead-letter rate

Queue length alone can be misleading because messages can have different processing costs.

Consumer Lag

In a log, consumer lag represents the distance between the log's latest available position and the consumer group's processed or committed position.

A simplified record-based definition is:

\[ ConsumerLag = LatestLogOffset - ConsumerCommittedOffset \]

Latest partition offset:

50,000


Consumer committed offset:

46,500


Consumer lag:

3,500 records

Record count should be combined with event age and processing rate because records can differ in size and processing cost.

Drain-time Estimate

A simplified backlog-drain estimate is:

\[ EstimatedDrainTime = \frac{ Backlog }{ ConsumerRate - ProducerRate } \]

This estimate is meaningful only when consumer rate is greater than producer rate. Retries, variation in message cost, partition imbalance, and dependency limits affect actual recovery.

Queue Use Cases

Log Use Cases

Decision Matrix

Combining Queues and Logs

A system does not need to select one messaging model for every workload.

Business transaction
      |
      v
Domain event appended to log
      |
      +-- Analytics consumer group
      +-- Search consumer group
      +-- Notification coordinator
              |
              v
        Notification queue
              |
              +-- Email worker
              +-- SMS worker
              +-- Push worker

The log distributes a retained business event to independent consumers. A downstream queue then dispatches concrete tasks to competing workers.

Event vs Command

Commands often fit queue semantics because one handler should perform the requested work. Events often fit log semantics because several independent consumers can react to the same fact.

Transactional Outbox

A service can update business data and record an outgoing message in the same local database transaction.

Application transaction
      |
      +-- Update business table
      +-- Insert outbox record
      |
      v
Commit once
      |
      v
Outbox publisher reads record
      |
      v
Publish to queue or log

The outbox publisher can retry publication. Consumers must still handle duplicate message delivery.

Conceptual Outbox Table

CREATE TABLE message_outbox
(
    message_id VARCHAR(150) PRIMARY KEY,
    aggregate_type VARCHAR(100) NOT NULL,
    aggregate_id VARCHAR(150) NOT NULL,
    message_type VARCHAR(150) NOT NULL,
    payload TEXT NOT NULL,
    created_at TIMESTAMP NOT NULL,
    published_at TIMESTAMP NULL
);

Example Queue Message

{
  "messageId": "unique-message-id",
  "commandType": "GenerateCourseCertificate",
  "tenantId": 17,
  "learnerId": 1042,
  "courseId": 42,
  "requestedAt": "message-created-time"
}

A command message should include a stable identifier, routing context, and only the data required by the consumer.

Example Log Event

{
  "eventId": "unique-event-id",
  "eventType": "CourseCompleted",
  "eventVersion": 1,
  "tenantId": 17,
  "learnerId": 1042,
  "courseId": 42,
  "occurredAt": "event-occurrence-time"
}

An event should describe an accepted fact. Event schemas should evolve compatibly because several independently deployed consumers can read the event.

Message Schema Evolution

Producers and consumers can be deployed at different times.

Use:

• Explicit message type and version
• Backward-compatible field additions
• Stable field meanings
• Consumer tolerance for unknown optional fields
• Controlled removal of obsolete fields
• Schema validation and compatibility testing

Do not reuse an existing field with a different meaning. Introduce a new version or field instead.

Security and Privacy

Queue and log records can outlive the original request and can be consumed by several systems.

Protect messaging data by:

• Publishing only the minimum required data
• Using trusted tenant and caller context
• Encrypting transport and storage according to policy
• Applying least-privilege producer and consumer access
• Separating tenants or workloads where required
• Protecting broker-management operations
• Redacting secrets and credentials
• Defining retention and deletion policies
• Auditing access to sensitive topics and queues

Event retention must be aligned with privacy and data-lifecycle requirements. Replay capability should not become indefinite retention without a defined purpose.

Conceptual Queue Configuration

queue:
  name: certificate-generation

  delivery:
    acknowledgment: explicit
    visibilityTimeout: approved-processing-boundary

  retries:
    maximumAttempts: approved-limit
    exponentialBackoff: true
    jitter: true

  deadLetter:
    enabled: true
    destination: certificate-generation-failures

  consumers:
    concurrency: approved-safe-limit

  observability:
    backlog: enabled
    oldestMessageAge: enabled
    retries: enabled
    deadLetters: enabled

Conceptual Log Configuration

eventLog:
  topic: learning-domain-events
  partitions: approved-partition-count
  replication: approved-replication-policy

  producer:
    key: trusted-aggregate-id
    idempotence: platform-supported

  retention:
    policy: approved-retention-policy

  consumers:
    groups:
      - notifications
      - search-index
      - analytics

  observability:
    partitionLag: enabled
    consumerGroupLag: enabled
    rebalanceEvents: enabled
    retainedStorage: enabled

These configurations are conceptual. Actual ordering, retention, acknowledgment, transactional, and delivery guarantees must be verified in the selected messaging platform's documentation.

PHP Idempotent Consumer Example

<?php

declare(strict_types=1);

final class EnrollmentEventConsumer
{
    public function __construct(
        private DatabaseConnection $database,
        private ProcessedMessageRepository $processedMessages,
        private EnrollmentProjection $projection
    ) {
    }

    public function consume(
        array $event
    ): void {
        $messageId =
            (string)$event['eventId'];

        $this->database->transaction(
            function () use (
                $messageId,
                $event
            ): void {
                if (
                    $this->processedMessages
                        ->exists(
                            consumerName:
                                'enrollment-projection',

                            messageId:
                                $messageId
                        )
                ) {
                    return;
                }

                $this->projection
                    ->applyEnrollmentEvent(
                        $event
                    );

                $this->processedMessages
                    ->record(
                        consumerName:
                            'enrollment-projection',

                        messageId:
                            $messageId
                    );
            }
        );
    }
}

Production code requires schema validation, secure tenant context, error classification, retry policy, database constraints, logging, and platform-specific acknowledgment or offset handling.

Learning-platform Examples

Observability

Queue Metrics

• Published-message rate
• Consumer completion rate
• Visible backlog
• Oldest-message age
• In-flight message count
• Redelivery rate
• Retry count
• Dead-letter count
• Processing duration
• Consumer concurrency

Log Metrics

• Records appended per partition
• Bytes appended per partition
• Consumer-group lag
• Oldest unprocessed-event age
• Partition traffic skew
• Consumer processing rate
• Offset-commit failures
• Consumer-group rebalances
• Retained storage
• Replication health

Structured Processing Event

{
  "messageSystem": "event-log",
  "destination": "learning-domain-events",
  "consumerGroup": "search-index",
  "partition": "protected-partition-reference",
  "processingResult": "success",
  "deliveryAttempt": 1,
  "schemaVersion": 1
}

Avoid logging message payloads when identifiers and processing metadata are sufficient.

Alert Conditions

Alert when:

• Queue backlog continues growing
• Oldest queued-message age exceeds the processing objective
• Consumer completion rate falls below producer rate
• Redelivery or retry rate increases unexpectedly
• Dead-letter volume grows
• One log partition becomes hot
• Consumer-group lag continues increasing
• Consumer-group rebalances occur repeatedly
• A poison event blocks partition progress
• Required log history approaches its retention boundary
• Broker storage approaches capacity
• Replication falls below its required level

Troubleshooting Workflow

1. Identify whether the destination uses queue or log semantics.
2. Confirm the message or event identity.
3. Check producer publication success.
4. Check broker availability and storage health.
5. For a queue, inspect backlog, in-flight messages, acknowledgments, and visibility.
6. For a log, inspect topic, partition, latest offset, and consumer-group offset.
7. Check consumer health and processing rate.
8. Check retries, redeliveries, and dead-letter handling.
9. Check partition or key skew.
10. Check downstream database and service dependencies.
11. Check idempotency and duplicate-processing records.
12. Check schema compatibility.
13. Check recent consumer-group or routing changes.
14. Replay or redrive messages through the approved recovery procedure.

Common Messaging Mistakes

Recommended Test Cases

Queues and Logs Best Practices

Practice Exercise

Design the messaging layer for your online learning platform.

Requirements

1. List every asynchronous workflow.
2. Classify each message as a command, event, or notification.
3. Use a queue for certificate-generation tasks.
4. Use a queue for course-thumbnail processing.
5. Publish course-completed facts to a retained event log.
6. Create separate notification, analytics, and search consumer groups.
7. Select a partition key preserving per-learner event order.
8. Add unique message and event identifiers.
9. Implement idempotent consumers.
10. Create bounded retries and dead-letter handling.
11. Use a transactional outbox for publication.
12. Monitor queue backlog and oldest-message age.
13. Monitor consumer lag by group and partition.
14. Test one poison message.
15. Replay events to rebuild a search projection.
16. Test consumer failure and group rebalancing.

Messaging-design Template

Frequently Asked Questions