Background Jobs Universal - Production Job Queue

Process background jobs across Celery, Bull, and AWS SQS with retries and scheduling.

Updated Jun 7, 2024
One-click install
npx skills add https://github.com/ashishgurprit/ANLP_A3 --skill background-jobs-universal-production-job-queue
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: Background Jobs Universal - Production Job Queue
Source: https://github.com/ashishgurprit/ANLP_A3/tree/main/.claude.backup/skills/background-jobs-universal
Command: npx skills add https://github.com/ashishgurprit/ANLP_A3 --skill background-jobs-universal-production-job-queue

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the need for robust and scalable background job processing in production environments, offering flexibility across different queuing technologies.

Core Features & Use Cases

  • Multi-Provider Support: Integrates with Celery (Python), Bull (Node.js), and AWS SQS for diverse backend needs.
  • Advanced Features: Includes retry logic with exponential backoff, scheduled jobs, priority queues, and dead-letter queues.
  • Use Case: Sending bulk emails, processing uploaded images, generating complex reports, or handling webhook events reliably.

Quick Start

Use the background jobs skill to enqueue a task to send an email to '[email protected]' with the subject 'Hello' and body 'World' using Celery.

Frequently Asked Questions about Background Jobs Universal - Production Job Queue

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up background jobs with retry logic and exponential backoff?

Background job processing with retry logic and exponential backoff is configured by defining task retry policies within your queue provider. This Skill handles failed task retries automatically, scaling wait times exponentially to prevent system overload during transient failures.

Can I use Celery, Bull, and AWS SQS interchangeably for asynchronous task execution?

Yes, this Skill provides a universal solution for asynchronous task execution across Celery, Bull, and AWS SQS. You can integrate with your preferred queuing technology to handle diverse backend requirements without changing core task definitions.

Do I need Redis to run background job processing with Celery and Bull?

Yes, Redis is required as a message broker when using Celery or Bull for background job processing. If you choose AWS SQS instead, you will need to configure valid AWS credentials rather than a Redis instance.

What is the best way to handle failed tasks in a production job queue?

The best way to handle failed tasks in a production job queue is using dead-letter queues. This Skill automatically routes messages that fail after maximum retry attempts to a dead-letter queue, preventing them from blocking active processing.

How do I schedule background jobs and configure priority queues?

Scheduling background jobs and configuring priority queues involves defining task execution times and priority levels within your queue setup. This Skill supports scheduled jobs and priority queuing to ensure critical tasks execute before standard operations.

What are the limitations of using background jobs for processing bulk emails and images?

Background job processing limitations depend on your chosen provider's throughput limits and broker capacity. While this Skill handles retries and dead-letter routing, processing bulk emails or images is constrained by your Redis or AWS SQS configuration limits.