dataqueue-advanced

Orchestrate DataQueue workflows with dependencies, memoization, waits, and cron.

24|1|Updated Jun 27, 2025

Search Tags:#workflow #dependencies #idempotency #cron #memoization #job-scheduling #dataqueue

Ready to run

npx skills add https://github.com/nicnocquee/dataqueue --skill dataqueue-advanced

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: dataqueue-advanced
Source: https://github.com/nicnocquee/dataqueue/tree/main/packages/dataqueue/ai/skills/dataqueue-advanced
Command: npx skills add https://github.com/nicnocquee/dataqueue --skill dataqueue-advanced

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Orchestrate advanced DataQueue workflows with dependencies, memoization, waits, and cron, ensuring reliable, lossless processing across distributed workers.

Core Features & Use Cases

Job dependencies with dependsOn to ensure prerequisites complete before continuation.
Step memoization using ctx.run to cache results and replay on re-invocation.
Time-based and token-based waits (waitFor, waitUntil, token waits) to coordinate human and system actions.
Cron scheduling and idempotent handling for recurring, fault-tolerant jobs.

Quick Start

Define a handler that uses ctx.run for memoization and ctx.waitFor for timing, then enqueue jobs with dependsOn, tags, cron, and idempotency settings to see advanced patterns in action.

Frequently Asked Questions about dataqueue-advanced

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I orchestrate job dependencies in a Node.js task queue?▼

Job dependencies are orchestrated using the dependsOn setting to ensure prerequisite tasks complete before continuation. This coordinates multi-step pipelines across distributed workers with strong guardrails and transactional job creation.

How does memoization work for replaying failed job steps in TypeScript?▼

Memoization works by using the ctx.run function to cache step results during execution. On re-invocation or retry, the system replays these cached results deterministically, preventing redundant processing and ensuring robust recovery.

Can I schedule recurring cron jobs with idempotent handling in PostgreSQL?▼

Yes, recurring cron jobs can be scheduled with idempotent handling to ensure fault-tolerant execution. This prevents duplicate processing of identical tasks across distributed workers while maintaining reliable, lossless scheduling.

What's the best way to coordinate human and system actions in a distributed workflow?▼

Time-based and token-based waits coordinate human and system actions by using waitFor, waitUntil, and token waits. These mechanisms pause pipeline execution until specific conditions or timeouts are met.

Does DataQueue support transactional job creation with Redis?▼

Yes, transactional job creation is supported for Node.js applications using Redis or PostgreSQL. This ensures lossless processing by atomically enqueuing jobs with dependencies, tags, and idempotency settings.

Why do my distributed workers process duplicate jobs during retries?▼

Duplicate jobs during retries occur without idempotent handling and deterministic replay. Implementing step memoization with ctx.run and idempotency settings ensures robust timeout semantics and prevents redundant processing.