python-background-jobs

Implement asynchronous task queues and background workers in Python using Celery.

Updated Apr 13, 2026
One-click install
npx skills add https://github.com/scoots31/engineering-playbook --skill python-background-jobs-scoots31
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: python-background-jobs
Source: https://github.com/scoots31/engineering-playbook/tree/main/references/python-background-jobs
Command: npx skills add https://github.com/scoots31/engineering-playbook --skill python-background-jobs-scoots31

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires celery, redis, fastapi, rq, dramatiq.

What problem does it solve? Long-running operations like sending emails, processing payments, or generating reports block request/response cycles and degrade user experience. This Skill provides patterns to decouple that work into background workers that process jobs asynchronously. ## Core Features & Use Cases - Task Queue Patterns: Return a job ID immediately while Celery workers process tasks in the background, with status polling endpoints for clients. - Reliability Patterns: Idempotent task design, exponential backoff retries, dead letter queues for failed jobs, and persisted job state machines. - Workflow Composition: Chain, group, and chord primitives for building multi-step pipelines and parallel fan-out processing. - Use Case: When a user requests a large data export, enqueue the job, return a job ID instantly, let a worker generate the file, and expose a /jobs/{id} endpoint so the client can poll for completion. ## Quick Start Set up a Celery background task that sends a welcome email asynchronously when a new user signs up, with retries and a job status endpoint.

Frequently Asked Questions about python-background-jobs

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run background tasks in Python with Celery?

Define a Celery app with a broker like Redis, decorate functions with @app.task, and call them with .delay() from your API handler. Workers running separately pick up and execute the tasks asynchronously.

Celery vs RQ vs Dramatiq for Python task queues?

Celery offers the most features including workflows, retries, and scheduling. RQ is simpler and Redis-only, good for basic queues. Dramatiq is a modern alternative with a cleaner API and reliable delivery defaults.

How do I make Celery tasks idempotent?

Check the current state before acting and return early if already processed. Pass idempotency keys to external services like payment providers, and use upsert patterns so retries never cause duplicate side effects.

What happens when a Celery task fails after all retries?

After max_retries is exhausted, the task should be routed to a dead letter queue with its payload, error, and attempt count. This preserves failed jobs for manual inspection instead of silently dropping them.

How do I let clients check background job status?

Persist job state transitions (pending, running, succeeded, failed) in a database and expose a GET /jobs/{job_id} endpoint. Return the status, timestamps, and result or error once the job reaches a terminal state.