kb-ingest-bulk

You can ingest large datasets by orchestrating a four-phase map-reduce workflow: parallel extraction, Python-based routing, per-file reduction, and shelf-index finalization. This coordinates extraction and synthesis across multiple libraries while maintaining resumable progress.

Handling bulk extraction failures is managed through resumable processing. The workflow supports retrying failed sources and continuing interrupted runs, maintaining a manifest and shelf index to track progress across large datasets without restarting from scratch.

Routing works by reading parallel extracts and assigning them to targets based on the existing library structure. A Python-based routing phase evaluates the extracted content and directs it to the appropriate per-file writers during the reduce phase.

Yes, you can process many sources in parallel. The map phase dispatches knowledge-extractor agents for each source simultaneously, and the workflow operates with no external dependencies, using preflight validations to ensure data integrity before processing begins.

A map-reduce workflow is needed when loading bulk knowledge-base sources that require coordinated extraction, routing, and synthesis across large datasets. It enables parallel processing, per-file consolidation, and resumable progress for interrupted or failed operations.

Limitations include processing overhead from orchestrating four distinct phases and the need for preflight validations. While it retries failed items, interrupted runs require resuming via the manifest, and routing depends on existing library structure alignment.