sysdesign-event-streaming-kafka

Choose Kafka over simpler queues when your event flow requires replay capabilities, strict per-key ordering, or multiple independent downstream consumers. If these requirements are absent, a simpler message broker is likely more appropriate.

Structure Kafka topics by creating one topic per bounded context, selecting appropriate partition keys for strict per-key ordering, and mapping downstream consumers to distinct consumer groups. Tailor your commit strategy and retention policies to support these specific design needs.

To support future backfills, configure Kafka retention policies to preserve historical event data and implement a strict schema management strategy. Documenting these governance policies ensures ongoing data compatibility and reliable replay capabilities.

You should not use Kafka for event streaming if your flow lacks replay requirements, strict per-key ordering needs, or multiple downstream consumers. In these cases, simpler queue systems provide adequate messaging without Kafka's operational overhead.

Handle failed messages in a Kafka topology by integrating a Dead Letter Queue (DLQ) for failure handling. Combine this with comprehensive observability and schema governance to monitor, manage, and recover from downstream consumer errors.