batch-observing

Parallel multi-user DM processing accelerates canvas creation by spawning per-user workers that run the ingest-dm pipeline simultaneously. A leader agent monitors progress, handles partial failures, and generates cross-canvas insights after ingestion.

Cross-canvas pattern detection analyzes multiple per-user canvases to identify shared insights across user bases. You need it after batch processing sizable DM exports or onboarding large groups, ensuring cross-user patterns are synthesized post-ingestion.

Handling partial failures during batch DM ingestion involves a leader agent that monitors worker progress and manages errors. It supports rate-limiting awareness and optional append-only updates for existing canvases, ensuring ingestion completes despite individual worker issues.

Batch DM ingestion is applicable when onboarding five or more new users. The leader agent spawns dedicated workers for each user, generating per-user canvases stored under grimoires/observer/canvas to streamline the onboarding workflow.

Orchestrating parallel workers for DM canvas generation requires a leader agent to spawn one worker per input DM export. Each worker independently runs the ingest-dm pipeline to process user data and generate individual canvases concurrently.