Stop 15 · Message Queues & Event-Driven ArchitectureTopics · 3/8

Apache Kafka Basics

In short

Kafka isn't a queue that forgets messages once read. It's a giant append-only log that many teams can read at their own pace, even days later.

A distributed event-streaming platform built for extremely high-throughput, durable, ordered event logs — used at large scale for event pipelines rather than simple task queues.

The blob writes at the end of a huge ledger book that has three ribbon bookmarks.

Think of it like this

The shop's ledger with bookmarks

An old shop writes every sale at the end of a thick ledger, and never tears pages out. The accountant reads it with one bookmark, the tax consultant with another, at his own speed. Kafka is that ledger, and each consumer group keeps its own bookmark.

Watch it happen

One partition of the orders topic

1/5Each cell is an event in one partition. The index is its offset, and new events are only ever added at the end.

The big ideas

What you need to know

Topics and partitions

A topic is split into partitions so it can spread across machines. Order is guaranteed within a partition, not across them.

Keys pick the partition

Events with the same key, like an orderId, always land in the same partition, so they stay in order.

Consumer groups

Within a group, each partition is read by one consumer. Different groups each read everything.

Replay

Events are kept for a set retention time. A new service can read from the start and rebuild its data.

See it in code

Walk through the code

bash

1# Create a topic split into 3 partitions2kafka-topics.sh --bootstrap-server localhost:9092 \3  --create --topic orders --partitions 34 5# Write events (one per line, typed as key:value)6kafka-console-producer.sh --bootstrap-server localhost:9092 --topic orders \7  --property parse.key=true --property key.separator=:8 9# Read as part of the "billing" consumer group10kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic orders \11  --group billing --from-beginning

1/3Three partitions means up to three consumers in one group can read in parallel.

--from-beginning only matters the first time a group reads. After that, the group resumes from its committed offset.

Save yourself the pain

Common mistakes

Oops

Expecting global order across a whole topic.

Do this instead

Kafka only orders within a partition. Use a key so related events share a partition.

Oops

Adding more consumers than partitions to go faster.

Do this instead

Extra consumers in a group sit idle. Partitions set the parallelism limit.

Your mission

See consumer groups split work

~40 min

Quick check

A topic has 4 partitions. You run 6 consumers in the same group. What happens?

Go deeper

1Read a high-level overview of Kafka's topic/partition/consumer-group model.
2Compare Kafka's log-based, replayable model to RabbitMQ's queue-based, consume-once model.
3Recognize Kafka is usually reached for at a scale most early-stage projects haven't hit yet.