Kafka vs RabbitMQ vs SQS

In a Nutshell

Three of the most common messaging systems represent three different philosophies. Apache Kafka is a distributed, durable log — it retains a replayable stream of events and excels at high-throughput data pipelines and event sourcing. RabbitMQ is a traditional message broker — a flexible, feature-rich router that excels at complex routing, per-message delivery, and task queues. Amazon SQS is a fully-managed, dead-simple cloud queue — zero operational overhead, near-infinite scale, minimal features. Choosing between them comes down to whether you need a replayable event log (Kafka), sophisticated routing and low-latency task distribution (RabbitMQ), or a hands-off managed queue (SQS).

How It Actually Works

The Three Philosophies

	Kafka	RabbitMQ	SQS
Model	Distributed commit log	Message broker (AMQP)	Managed cloud queue
Message retention	Retained (days/forever); replayable	Deleted after ack	Deleted after ack (max 14 days)
Consumption	Consumers track offsets, can replay	Broker pushes; message gone once acked	Poll-based; message gone once deleted
Ordering	Per-partition ordering	Per-queue (best-effort)	FIFO queues (optional) or best-effort
Throughput	Very high (millions/sec)	High (tens of thousands/sec)	High (managed, elastic)
Routing	Simple (topic/partition)	Rich (exchanges, bindings, routing keys)	Minimal
Ops overhead	High (run a cluster) or managed (MSK)	Medium (run a broker)	None (fully managed)
Best at	Streaming, event sourcing, pipelines	Complex routing, task queues, RPC	Simple decoupling on AWS

Kafka: The Replayable Log

Kafka's defining trait is that it's a durable, append-only log, not a queue that deletes messages. Consumers read at their own offset and messages stay put:

Topic "orders" (partitioned):
  Partition 0: [m0][m1][m2][m3][m4][m5]...  ← messages appended, never deleted
                       ▲         ▲
              Consumer-A     Consumer-B      each reads at its OWN offset
   → Consumers can rewind and REPLAY history
   → New consumer can read from the beginning
   → Multiple independent consumer groups read the same stream

This makes Kafka ideal for: event sourcing, stream processing, feeding multiple downstream systems from one source of truth, and reprocessing data after a bug fix. Partitions provide ordering (within a partition) and parallelism (across partitions).

RabbitMQ: The Smart Router

RabbitMQ's strength is flexible routing via exchanges. Producers publish to an exchange, which routes to queues based on rules:

Exchange Type	Routes By	Use
Direct	Exact routing key match	Point-to-point by key
Topic	Wildcard pattern (order.*.eu)	Selective subscription
Fanout	To all bound queues	Broadcast
Headers	Message header attributes	Attribute-based routing

RabbitMQ pushes messages to consumers (low latency), supports priorities, TTLs, and per-message acknowledgment, and shines for task queues and complex enterprise routing. It deletes messages once acked (no replay).

SQS: The Managed Simplicity

SQS trades features for zero operations. There are no servers to manage, it scales elastically, and it just works. Two flavors:

• Standard — near-unlimited throughput, at-least-once, best-effort ordering.
• FIFO — exactly-once processing and strict ordering, at lower throughput.

It has minimal routing and no replay, but for "decouple these two services on AWS without running infrastructure," it's often the right, boring choice.

The Decision Framework

Need to REPLAY events / stream processing / one source → many consumers?
   → Kafka

Need COMPLEX ROUTING, priorities, or low-latency task queues,
and you can run/manage a broker?
   → RabbitMQ

Want ZERO ops, simple decoupling, already on AWS?
   → SQS

Seeing It in Action

Scenario: Picking the right tool for three different needs in one company.

Need 1 — Real-time analytics + event sourcing:
  "Every user click, order, and page view must feed analytics,
   ML models, a data warehouse, AND be replayable after bugs."
  → KAFKA.
    One "events" topic; multiple consumer groups (analytics, ML,
    warehouse) each read independently at their own offset. When the
    ML pipeline has a bug, replay the last 7 days from the log.
    Millions of events/sec across partitions.

Need 2 — Background job processing with priorities:
  "Image-processing jobs; premium users' jobs go first; failed jobs
   retry then dead-letter; some jobs route to GPU workers."
  → RABBITMQ.
    Priority queues put premium jobs first; a topic exchange routes
    'job.gpu.*' to GPU-worker queues and 'job.cpu.*' elsewhere;
    per-message acks + DLQ handle failures. Rich routing is the win.

Need 3 — Decouple order service from email service (on AWS):
  "When an order is placed, send a confirmation email. Simple.
   Don't want to run a broker."
  → SQS.
    Order service enqueues; an email Lambda drains it. Fully managed,
    scales automatically, near-zero ops. Boring and correct.

The meta-lesson: these tools aren't strictly competitors — mature systems often use all three for different jobs. Kafka is a data-streaming backbone, RabbitMQ is a task-routing workhorse, and SQS is the effortless glue between services. Match the tool to the shape of the problem: replayable stream, routed tasks, or simple managed decoupling.

Interview Questions

1. Q: What fundamentally distinguishes Kafka from RabbitMQ and SQS? Hint: Kafka is a durable, replayable append-only log — messages are retained and consumers read at their own offset, so multiple independent consumers can read (and replay) the same stream. RabbitMQ and SQS are queues that delete messages once acknowledged (no replay). Kafka suits event streaming/sourcing and one-source-to-many-consumers; the others suit transient work distribution.

2. Q: When would you choose RabbitMQ over Kafka? Hint: When you need sophisticated routing (direct/topic/fanout/headers exchanges), message priorities, per-message TTLs, low-latency push-based task queues, or request/reply patterns — and you don't need to replay history. RabbitMQ is a feature-rich broker optimized for complex routing and task distribution, whereas Kafka is optimized for high-throughput retained streams with simple topic/partition routing.

3. Q: What do Kafka partitions provide, and what's the ordering guarantee? Hint: Partitions provide parallelism (consumers scale across partitions) and ordering within a partition (messages in one partition are strictly ordered). There's no global ordering across partitions. Messages with the same key go to the same partition, preserving per-key order. More partitions = more parallelism but weaker global ordering.

4. Q: Why might you choose SQS despite its minimal feature set? Hint: Zero operational overhead — no cluster to run, patch, or scale; it's fully managed and elastically scalable. For simple service decoupling on AWS, that operational simplicity outweighs missing features like replay or rich routing. SQS Standard gives near-infinite throughput (at-least-once, best-effort order); SQS FIFO gives strict ordering and exactly-once at lower throughput.

5. Q: Can these systems be used together? Give an example. Hint: Yes — they're complementary. A company might use Kafka as the event-streaming backbone (analytics, ML, warehouse, replay), RabbitMQ for priority-routed background job processing, and SQS to simply decouple two services on AWS. Match each to the problem shape: replayable stream (Kafka), routed/prioritized tasks (RabbitMQ), managed glue (SQS).

References

• Kafka: The Definitive Guide — the durable-log model in depth
• RabbitMQ documentation — exchanges, routing, and delivery
• AWS SQS documentation — Standard vs FIFO

Dive Deeper

• Kafka vs RabbitMQ (Confluent) — detailed architectural comparison
• The Log (Jay Kreps) — why the log model is powerful
• Uber: choosing messaging systems at scale — real-world messaging trade-offs