upstash-workflow

Use a three-layer architecture: a process-* entry point that validates and filters items, a paginate-* layer that handles cursor pagination and fan-out, and an execute-* layer that processes exactly one item per invocation. Each layer is a separate Next.js route using serve() from @upstash/workflow/nextjs.

Use the fan-out pattern in the pagination layer: when a page exceeds CHUNK_SIZE (default 20), split it into chunks and recursively re-trigger the paginate route with each chunk's itemIds. This distributes work across multiple workflow invocations instead of one oversized run.

Upstash deduplicates steps by name, so reusing the same step name inside Promise.all causes data loss. Give each step a unique name like `workflow:execute:${item.id}` so every item's result is recorded separately.

Set the entry point to parallelism 1 and ratePerSecond 1 to prevent duplicate batch triggers, pagination to parallelism 10-20, and execution to parallelism 5-10 depending on downstream API rate limits. Each layer uses a distinct flowControl key.

Yes, create re-export route files in the cloud project that export POST from the lobehub/src path rather than the @/ alias. Using the @/ alias causes circular import errors, while the direct package path resolves correctly.

Trigger the entry point with dryRun: true to get statistics on eligible and already-processed items without side effects. Then run a small batch, verify idempotent filtering skips processed items, and write integration tests covering both the dry-run and full execution paths.