Introduction

Distributed applications frequently need to perform work outside the original request-response path.

Client submits request
      |
      v
Application validates request
      |
      v
Application publishes message
      |
      v
Application responds
      |
      v
Consumer processes message asynchronously

Apache Kafka, RabbitMQ, and Amazon Simple Queue Service, commonly called Amazon SQS, all help producers communicate asynchronously with consumers. However, these technologies are based on different messaging models.

• Apache Kafka is primarily a distributed event-streaming log.
• RabbitMQ is a message broker built around exchanges, queues, bindings, and acknowledgments.
• Amazon SQS is a fully managed AWS queue service offering Standard and FIFO queues.

Prerequisites

Quick Comparison

Apache Kafka

Apache Kafka organizes records into topics. Each topic is divided into one or more partitions, and each partition is an ordered append-only log.

Kafka topic:

learning-domain-events


Partition 0:

Offset 0 -> CourseCreated
Offset 1 -> CoursePublished
Offset 2 -> CourseArchived


Partition 1:

Offset 0 -> LearnerEnrolled
Offset 1 -> LessonCompleted
Offset 2 -> CourseCompleted

Consumers read records using offsets. Reading a record does not remove it from the topic.

Main Kafka Components

Kafka Write Path

Producer creates event
      |
      v
Select topic
      |
      v
Select partition using key
or partitioning policy
      |
      v
Kafka appends record
to partition log
      |
      v
Record is replicated
according to cluster policy
      |
      v
Producer receives result
according to acknowledgment policy

Kafka Read Path

Consumer joins group
      |
      v
Partitions are assigned
      |
      v
Consumer fetches records
from current offsets
      |
      v
Consumer processes records
      |
      v
Consumer commits progress

Kafka Consumer Groups

Consumers inside one group divide topic partitions. Separate groups consume the same retained stream independently.

Topic:

Partition 0
Partition 1
Partition 2
Partition 3


Notification Group:

Consumer N1 -> Partitions 0 and 1
Consumer N2 -> Partitions 2 and 3


Analytics Group:

Consumer A1 -> Partition 0
Consumer A2 -> Partition 1
Consumer A3 -> Partition 2
Consumer A4 -> Partition 3

A partition is assigned to one active consumer within a group at a given time. Adding consumers beyond the group's available partitions does not create additional active partition ownership.

Kafka Offsets and Replay

Kafka consumers commit offsets so that processing can resume after restart or reassignment.

Partition latest offset:

25,000


Committed group offset:

23,500


Consumer lag:

1,500 records

A consumer can replay retained records by beginning from an earlier valid offset.

Current offset:

25,000


Replay offset:

20,000


Consumer reprocesses:

20,000 through 25,000

Replay is useful for:

• Rebuilding a search index
• Recreating a materialized view
• Correcting a previously defective consumer
• Feeding a new downstream system
• Reprocessing analytical data

Kafka Ordering

Kafka provides ordering inside an individual partition.

Partition for learner-1042:

Offset 100 -> CourseStarted
Offset 101 -> LessonCompleted
Offset 102 -> QuizPassed
Offset 103 -> CourseCompleted

Kafka does not automatically provide one global order across all partitions. Events requiring local ordering should use a stable key that places them in the same partition.

Kafka Hot Partitions

A low-cardinality or highly popular record key can overload one partition.

Normal keys:

1,000 events per minute


One popular key:

500,000 events per minute


Result:

The partition owning the key
becomes the bottleneck.

Possible controls include:

• Choose a higher-cardinality partition key
• Split safely independent work into subkeys
• Create a controlled bucket component
• Separate very heavy workloads into another topic
• Apply producer rate limits
• Add partitions before current partition capacity is exhausted

Kafka Connect

Kafka Connect provides a framework for transferring data between Kafka and supported external systems using source and sink connectors.

Source system
      |
      v
Source connector
      |
      v
Kafka topic
      |
      v
Sink connector
      |
      v
Target system

Source connectors can bring records into Kafka. Sink connectors can consume Kafka records and transfer them to external systems such as databases, search indexes, files, or HTTP endpoints, depending on the connector.

Kafka Strengths

• Retained and replayable event streams
• Independent consumer groups
• Partition-based parallelism
• Ordering within a partition
• Strong fit for Change Data Capture and stream processing
• Suitable foundation for multiple real-time downstream pipelines
• Connector ecosystem for supported source and target systems

Kafka Trade-offs

• Topics, partitions, retention, replication, and consumer groups require operational design
• Consumer-group rebalancing must be handled correctly
• Partition-count and partition-key choices affect throughput and ordering
• Hot partitions can limit parallelism
• Kafka is usually more complex than a simple managed task queue
• Task priority and complex per-message routing can require additional design

RabbitMQ

RabbitMQ is a message broker in which producers commonly publish to an exchange. The exchange routes messages to one or more queues using its type, bindings, and routing information.

Producer
    |
    v
Exchange
    |
    +-- Queue A -> Consumer Group A
    +-- Queue B -> Consumer Group B
    +-- Queue C -> Consumer Group C

Main RabbitMQ Components

RabbitMQ Exchange Types

Topic-routing Example

Message routing key:

learning.course.published


Binding:

learning.course.*


Result:

Message matches the binding
and is routed to the queue.

RabbitMQ Acknowledgments

Consumer acknowledgments tell RabbitMQ when a delivery has been processed successfully.

RabbitMQ delivers message
      |
      v
Consumer processes message
      |
      +-- Success:
      |      send acknowledgment
      |      message can be removed
      |
      +-- Failure:
             reject or negatively acknowledge
             according to retry policy

Publisher confirms and consumer acknowledgments solve different parts of the delivery path.

• Publisher confirms address publication from publisher to RabbitMQ.
• Consumer acknowledgments address processing from RabbitMQ to consumer.

RabbitMQ Prefetch

Prefetch limits the number of unacknowledged deliveries sent to a consumer or channel, according to the client and broker configuration.

Consumer prefetch:

10


Maximum assigned but
unacknowledged deliveries:

10

A very large prefetch can overload a slow consumer and distribute work unevenly. A very small prefetch can reduce throughput for fast consumers. Tune it using measured processing duration and memory consumption.

RabbitMQ Competing Consumers

Task Queue
   |
   +-- Worker A
   +-- Worker B
   +-- Worker C

Consumers connected to one queue compete for its messages. If several independent services need the same published message, the exchange can route copies to separate queues.

RabbitMQ Strengths

• Flexible exchange and binding model
• Strong fit for task and worker queues
• Competing-consumer processing
• Publisher confirms and consumer acknowledgments
• Suitable for direct, topic, fanout, and header-based routing
• Useful when the broker should make routing decisions

RabbitMQ Trade-offs

• Traditional queues are not primarily designed as retained event history
• Each independent subscriber commonly needs its own queue
• Broker topology and queue lifecycle require active management
• Incorrect acknowledgment or requeue behaviour can cause loss or repeated delivery
• Poison messages can create repeated redelivery
• Applications must tune prefetch and consumer concurrency

Amazon SQS

Amazon SQS is a managed AWS queue service. Producers send messages to a queue, and consumers poll the queue for available messages.

Producer
    |
    v
Amazon SQS Queue
    |
    +-- Consumer A
    +-- Consumer B
    +-- Consumer C

Amazon SQS provides two queue types:

• Standard queues
• FIFO queues

SQS Standard Queues

Standard queues are designed for very high throughput, at-least-once delivery, and best-effort ordering.

Applications using Standard queues must handle:

• Occasional duplicate delivery
• Messages arriving in a different order
• Idempotent processing
• Visibility timeout expiration
• Explicit message deletion after success

SQS FIFO Queues

FIFO queues are designed for workloads where ordering and duplicate suppression within the queue's documented processing model are important.

FIFO queues use message groups to provide separate ordered sequences inside one queue.

FIFO Queue


Message Group: learner-1042

RegisterAccount
EnrollInCourse
CompleteCourse


Message Group: learner-2055

RegisterAccount
EnrollInCourse
CompleteCourse

Messages inside one group are processed in order. Different message groups can support parallel progress.

SQS Standard vs FIFO

SQS Visibility Timeout

When a consumer receives an SQS message, the message becomes temporarily invisible to other consumers.

Consumer receives message
      |
      v
Visibility timeout begins
      |
      +-- Processing succeeds:
      |      consumer deletes message
      |
      +-- Message is not deleted:
             visibility timeout expires
             message becomes available again

If the visibility timeout is shorter than normal processing duration, another consumer can receive the same message before the first consumer finishes.

If the visibility timeout is unnecessarily long, failed work can take longer to become eligible for retry.

SQS Delete after Success

Receiving a message does not mean that processing has completed. The consumer should delete the message only after the required business outcome has completed successfully.

Receive message
      |
      v
Validate payload
      |
      v
Process idempotently
      |
      v
Commit business outcome
      |
      v
Delete message

SQS Dead-letter Queues

An SQS queue can use a dead-letter queue for messages that are not processed successfully after the configured receive attempts.

Source Queue
      |
      v
Consumer processing fails
      |
      v
Message becomes visible again
      |
      v
Receive count reaches policy
      |
      v
Move to Dead-letter Queue

A dead-letter workflow should include monitoring, investigation, correction, and controlled redrive.

SQS Strengths

• Managed AWS queue infrastructure
• Standard and FIFO queue options
• Visibility timeout for in-progress work
• Dead-letter queue support
• Strong fit for AWS-native background processing
• Consumers can scale independently from producers
• No messaging-broker cluster administration by the application team

SQS Trade-offs

• It is a queue service rather than a replayable event-streaming log
• Consumers poll queues rather than reading retained partition logs
• Standard messages can be duplicated or delivered out of order
• FIFO parallelism depends on appropriate message-group design
• Complex fanout and routing can require additional AWS services or queues
• Applications must tune visibility, retries, deletion, and dead-letter behaviour

Detailed Feature Comparison

Choosing the Correct Platform

Choose Kafka When

• Events must remain replayable
• Several independent consumer groups require the same event stream
• Change Data Capture is a major requirement
• Real-time stream processing is required
• Ordering is needed within a stable business key
• Historical events must rebuild projections or indexes
• Event-stream retention is part of the architecture

Choose RabbitMQ When

• Messages require flexible exchange-based routing
• Commands must be distributed among competing workers
• Direct, topic, fanout, or header-based routing is useful
• Per-message acknowledgment and requeue behaviour are important
• The team can operate RabbitMQ or use an appropriate managed offering
• Retained event replay is not the central requirement

Choose Amazon SQS When

• The solution runs primarily on AWS
• A managed queue is preferred over operating brokers
• The workload is background-task processing
• Producer and consumer scaling should remain independent
• Standard queue duplicates and reordering can be handled
• FIFO message groups meet the required ordering scope
• Visibility timeout and dead-letter queue semantics fit the workflow

Using More than One Platform

One organization can use different messaging technologies for different workloads.

Business transaction
      |
      v
Kafka event:

CourseCompleted
      |
      +-- Analytics consumer group
      +-- Search consumer group
      +-- Certification service
              |
              v
        RabbitMQ or SQS task:

        GenerateCertificate
              |
              v
        Competing workers

Kafka distributes and retains the business fact. RabbitMQ or SQS can then dispatch a concrete unit of work.

Command and Event Examples

Delivery and Idempotency

All three systems require application-level attention to duplicate processing and uncertain outcomes.

Consumer performs business update
      |
      v
Update commits
      |
      v
Acknowledgment, offset commit,
or delete request fails
      |
      v
Message can be processed again

Use:

• Stable message or event identifiers
• Unique database constraints
• Conditional writes
• Processed-message records
• Idempotency keys
• Transactional consumer updates where supported

Idempotent Message

{
  "messageId": "stable-unique-message-id",
  "messageType": "GenerateCertificate",
  "messageVersion": 1,
  "tenantId": 17,
  "learnerId": 1042,
  "courseId": 42
}

Transactional Outbox

A transactional outbox helps prevent a service from committing business data without recording the corresponding message.

Local database transaction
      |
      +-- Update business record
      +-- Insert outbox message
      |
      v
Commit once
      |
      v
Publisher reads outbox
      |
      v
Publish to Kafka,
RabbitMQ, or SQS
      |
      v
Mark publication progress

The publisher can retry after failure. Consumers must still be idempotent because publication can occur more than once.

Backpressure

A messaging platform buffers work, but buffering does not create unlimited consumer capacity.

Let \(P\) be the production rate and \(C\) be the successful consumer completion rate.

Backlog grows when:

\[ P > C \]

A simplified backlog growth rate is:

\[ BacklogGrowthRate = P - C \]

Use:

• Producer rate limits
• Bounded consumer concurrency
• Autoscaling based on backlog or lag
• Batching where supported and appropriate
• Load shedding for optional work
• Dead-letter handling for poison messages

Conceptual Kafka Configuration

kafka:
  topic: learning-domain-events
  partitions: approved-partition-count
  replication: approved-replication-policy

  producer:
    key: trusted-aggregate-id
    acknowledgments: approved-durability-policy

  consumer:
    groupId: analytics-projection
    offsetCommit: after-durable-processing

  retention:
    policy: approved-replay-requirement

  observability:
    consumerLag: enabled
    partitionSkew: enabled
    rebalanceEvents: enabled
    retainedStorage: enabled

Conceptual RabbitMQ Configuration

rabbitmq:
  exchange:
    name: learning.commands
    type: topic

  queue:
    name: certificate-generation
    durable: true

  binding:
    routingKey: certificate.generate

  publisher:
    confirms: enabled

  consumer:
    acknowledgment: manual
    prefetch: approved-safe-limit

  failureHandling:
    boundedRetries: true
    deadLetterDestination: certificate-failures

  observability:
    readyMessages: enabled
    unacknowledgedMessages: enabled
    redeliveries: enabled

Conceptual SQS Configuration

sqs:
  queueType: standard-or-fifo
  queueName: certificate-generation

  processing:
    visibilityTimeout: approved-processing-boundary
    deleteAfterDurableSuccess: true
    idempotentConsumerRequired: true

  fifo:
    messageGroupKey: trusted-business-key
    deduplicationKey: stable-message-id

  failureHandling:
    deadLetterQueue: certificate-generation-failures
    maximumReceives: approved-limit

  observability:
    availableMessages: enabled
    inFlightMessages: enabled
    oldestMessageAge: enabled
    deadLetterMessages: enabled

These are conceptual examples. Exact settings and supported guarantees must be verified against the deployed platform and version.

PHP Idempotent Consumer

<?php

declare(strict_types=1);

final class CertificateCommandHandler
{
    public function __construct(
        private DatabaseConnection $database,
        private ProcessedMessageRepository $processedMessages,
        private CertificateService $certificates
    ) {
    }

    public function handle(
        array $message
    ): void {
        $messageId =
            (string)$message['messageId'];

        $this->database->transaction(
            function () use (
                $messageId,
                $message
            ): void {
                if (
                    $this->processedMessages
                        ->exists(
                            consumerName:
                                'certificate-command-handler',

                            messageId:
                                $messageId
                        )
                ) {
                    return;
                }

                $this->certificates
                    ->generateIfNotExists(
                        tenantId:
                            (int)$message['tenantId'],

                        learnerId:
                            (int)$message['learnerId'],

                        courseId:
                            (int)$message['courseId']
                    );

                $this->processedMessages
                    ->record(
                        consumerName:
                            'certificate-command-handler',

                        messageId:
                            $messageId
                    );
            }
        );
    }
}

Broker acknowledgment, offset commit, or SQS deletion should happen only after this durable processing boundary succeeds.

Security Considerations

• Grant producers access only to required topics, exchanges, or queues.
• Grant consumers access only to their required destinations.
• Encrypt messaging traffic and stored messages according to policy.
• Do not publish passwords, access tokens, or reusable credentials.
• Use trusted tenant context rather than client-supplied routing identity.
• Protect broker-management and queue-management operations.
• Define event retention according to privacy and business requirements.
• Redact sensitive payloads from processing logs and dead-letter diagnostics.
• Audit administrative changes and message-access permissions.

Observability

Kafka Metrics

• Records produced by topic and partition
• Consumer-group lag
• Oldest unprocessed-event age
• Partition traffic skew
• Consumer processing rate
• Offset-commit failures
• Consumer-group rebalances
• Broker storage and replication health

RabbitMQ Metrics

• Ready-message count
• Unacknowledged-message count
• Publish and delivery rates
• Consumer acknowledgment rate
• Redelivery rate
• Queue depth and message age
• Dead-letter volume
• Consumer and connection health

SQS Metrics

• Available-message count
• In-flight-message count
• Age of oldest message
• Messages sent, received, and deleted
• Receive count and retry behaviour
• Dead-letter queue depth
• Consumer completion rate
• Visibility-timeout expirations inferred from redelivery

Alert Conditions

Alert when:

• Kafka consumer lag continues increasing
• One Kafka partition receives disproportionate traffic
• Kafka consumers rebalance repeatedly
• RabbitMQ ready or unacknowledged messages continue growing
• RabbitMQ redelivery volume increases unexpectedly
• SQS oldest-message age exceeds the processing objective
• SQS messages repeatedly return after visibility timeout
• A dead-letter queue receives unexpected messages
• Consumer completion rate remains below producer rate
• Poison messages block useful processing
• Messaging storage or broker resources approach capacity
• Schema compatibility errors increase

Common Selection Mistakes

Recommended Test Cases

Best Practices

Practice Exercise

Select Kafka, RabbitMQ, or Amazon SQS for the asynchronous workflows in your online learning platform.

Requirements

1. Publish learner-domain events to Kafka.
2. Create independent event consumers for analytics and search.
3. Use learner ID as the Kafka key where per-learner ordering is required.
4. Replay retained events to rebuild a search index.
5. Create a RabbitMQ topic exchange for complex notification routing.
6. Create separate RabbitMQ queues for email and push notifications.
7. Configure manual acknowledgments and bounded prefetch.
8. Use SQS for AWS-native course-report generation.
9. Compare SQS Standard and FIFO for the report workflow.
10. Create an SQS dead-letter queue.
11. Add idempotency to every consumer.
12. Use a transactional outbox for event and command publication.
13. Test poison-message handling on all selected platforms.
14. Monitor Kafka lag, RabbitMQ queue depth, and SQS oldest-message age.
15. Document why each workflow uses its selected platform.

Selection Template

Frequently Asked Questions