message-queue

Choose Kafka for high throughput above 100k messages per second and message replay, RabbitMQ for complex routing with topic exchanges, and SQS for simple serverless queues on AWS. The decision matrix compares them across delivery guarantees, fan-out, and managed hosting options.

Declare a dead letter exchange and bind a quorum queue to it, then set x-dead-letter-exchange on the main queue. Failed messages are routed to the DLQ after retries are exhausted, and you should alert whenever the DLQ contains messages.

Include a unique event_id in every message and store processed IDs in Redis with a TTL. Before handling an event, check if its ID was already processed and skip duplicates, since at-least-once delivery makes redelivery inevitable.

SQS FIFO queues support deduplication through content-based deduplication or explicit deduplication IDs, which approximates exactly-once within the deduplication window. Consumers should still be idempotent because duplicates can occur outside that window.

Consumer lag grows when processing is slower than production or partitions outnumber consumer instances. Monitor consumer group lag as the primary SLO, increase partitions for parallelism, and avoid slow synchronous calls inside the message handler.

Kafka is overkill for simple single-consumer task queues where RabbitMQ or SQS is a better fit. It also has limited routing flexibility compared to RabbitMQ topic exchanges and requires more operational overhead.