Design brief
Design a retained log for order events at 50,000 events per second, averaging 1 KB and retained for 24 hours. Logical payload is about 4.32 TB daily; three copies require roughly 13 TB before overhead. Preserve order within an order while allowing unrelated orders to progress independently. Support independent consumer groups and replay. Define producer durability and consumer recovery promises.
Draw and explain
Use Excalidraw to show partition placement, replicated storage, producer routing, group coordination, and downstream business effects. Label leaders, ownership generations, offsets, and durable progress. Trace append through acknowledgement, consumption through progress advancement, and a new group replaying retained data. Estimate storage and recovery bandwidth, then explain how partition count limits useful consumer parallelism.
Stress the design
A consumer finishes offset 11 before offset 10, then crashes. Draw its valid restart point and which records may repeat. Next, reassign a partition while its old consumer pauses and later resumes. Explain how stale progress updates are rejected and how an external charge avoids duplication if processing repeats.
Review your design
Check ordered effects separately from ordered log records. Compare replication acknowledgements against write latency and availability. Explain hot-order limits, poison-event handling, and what happens when lag exceeds retention. Assess the cost of repartitioning and group rebalances. Choose signals for replication health, acknowledgement latency, lag age, and replay capacity.
Capture assumptions, sizing, and the tradeoffs behind your diagram.
Review is your own assessment of the design.