async-patterns

Choose Redis for sub-millisecond latency with simple tasks, SQS for managed durability and reliability, Kafka for high throughput and ordered processing per partition, and RabbitMQ for complex routing with exchanges. Match the queue to your latency, persistence, and ordering requirements.

Use the retry method with a countdown calculated as 2 raised to the retry count, producing delays of 2, 4, 8, 16 seconds. Set max_retries to cap attempts and route permanently failed tasks to a dead letter queue for manual investigation.

Idempotency ensures a task produces the same result when executed multiple times, preventing duplicate side effects like double charges after retries. Implement it with unique idempotency keys stored in the database or enforced via unique constraints.

Use Kafka when you need ordered processing per entity via partitions, throughput above 100k messages per second, or event replay capabilities. SQS is simpler and fully managed but lacks partition-level ordering except in FIFO mode.

Use Kubernetes HorizontalPodAutoscaler with an external metric like celery_queue_length, scaling replicas between a minimum and maximum when messages per worker exceed a threshold. AWS Lambda scales automatically based on SQS queue depth.

Duplicate processing happens when retries occur after partial completion or when at-least-once queues redeliver messages. Add idempotency keys, use database unique constraints, and commit message offsets only after successful processing.