Kafka

Creating an order used to finish after one database transaction. Now checkout also calls email, analytics, and a warehouse loader. If one of those services slows down, the customer waits for work that could happen later. Adding another consumer means changing checkout again.

Kafka lets the application publish an event to a retained log, which other services consume independently. LinkedIn built it for high-volume activity streams and open-sourced it in 2011. Jay Kreps, Neha Narkhede, and Jun Rao described the original system: one pipeline could feed both live processing and offline analysis.

Events belong to topics, and each topic is split into partitions. Records are ordered within a partition and identified there by an offset. Supplying an order ID as the key can keep that order's events together, provided the partitioning arrangement stays consistent. Kafka does not provide one global order across all partitions.

In an Apache Kafka distribution directory, with a local broker already listening on port 9092, create an example topic:

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic profile-orders --partitions 3 --replication-factor 1

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --describe --topic profile-orders

One replica suits this single-broker example and provides no replica failover. A replicated deployment needs additional brokers and deliberate producer acknowledgement and in-sync-replica settings.

Publish a keyed event. The console producer below treats the text before the first colon as the key and the remaining JSON as the value:

printf '%s\n' '42:{"order_id":42,"status":"created"}' |
  bin/kafka-console-producer.sh --bootstrap-server localhost:9092 \
    --topic profile-orders \
    --property parse.key=true --property key.separator=:

Kafka stores batches in partition logs, making append-oriented I/O and sequential consumption useful to its throughput. Batching amortizes network and storage overhead; the operating system's page cache helps serve active data. Throughput comes from those choices and partition-level parallelism, rather than from avoiding disks entirely.

The warehouse loader can read through its own consumer group:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
  --topic profile-orders --group profile-warehouse \
  --from-beginning --property print.key=true

Stop the console consumer with Ctrl-C. Within a conventional consumer group, each partition is assigned to one consumer at a time. Another group, such as profile-email, has independent offsets and can read the same events without taking them away from the warehouse. --from-beginning applies when the group has no valid committed offset; reusing a group normally resumes its saved position.

Inspect how far the group has committed:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group profile-warehouse

Offsets describe progress through the log, not whether a business effect happened exactly once. If a worker writes to the warehouse and crashes before committing its offset, it may process that record again. Use stable event IDs and idempotent writes. Retention also bounds replay: a lagging consumer can lose access to records that have aged out.

Checkout still needs to bridge its database transaction and event publication. Writing an outbox row with the order, then publishing it through a relay or change-data pipeline, avoids a gap between committing the order and sending its event. The same log can then support analytics, notifications, and rebuilding derived views while each consumer moves at its own pace.

Queues, streams, and transactional messaging follows that database-to-event boundary in more detail.