python-background-jobs

Define a Celery app with a broker like Redis, decorate functions with @app.task, and call them with .delay() from your API handler. Workers execute the tasks asynchronously while your API returns immediately.

Celery offers the most features including task chaining, chords, and groups for complex workflows. RQ is simpler and Redis-only, while Dramatiq provides a modern API with sensible defaults. Cloud options like AWS SQS or Google Cloud Tasks suit serverless deployments.

Check the current state before acting and return early if already processed. Use idempotency keys when calling external services like payment providers, and apply upsert patterns or deduplication windows to handle at-least-once delivery safely.

Route permanently failed tasks to a dead letter queue by checking self.request.retries against max_retries in the exception handler. Send the task payload, error details, and attempt count to the DLQ for manual inspection instead of retrying forever.

No. Retry only transient errors like connection timeouts with exponential backoff. Permanent failures such as validation errors, declined payments, or invalid credentials should return immediately without retrying.