event_driven

Start by modeling canonical event schemas with a versioning strategy, then choose between choreography (pub/sub) and orchestration (saga orchestrator) per data flow. Select a broker like Kafka or RabbitMQ, and plan failure handling with Dead-Letter Queues and idempotent consumers.

Select the broker based on load and latency needs: Kafka suits high-throughput event streaming with partitioning and retention control, RabbitMQ fits traditional message queuing, and SQS works for managed cloud queues. Configure ordering, partitions, and retention policies accordingly.

Add a unique event_id to every event and build idempotent consumers that safely process the same event twice. Combine this with automated retry logic and Dead-Letter Queues so failed messages can be replayed without corrupting state.

Avoid it for simple request-response workflows where loose coupling adds no value, since it introduces operational complexity in tracing and debugging. Without strong observability like distributed tracing and consumer lag alerts, diagnosing stuck consumers becomes very difficult.

Instrument the full chain from producer through broker to consumer using OpenTelemetry distributed tracing. Track consumer lag, message throughput, schema validation failures, and Dead-Letter Queue depth with dashboards and alarms.