kb-ingest-batch

Batch ingesting markdown files with resume support involves staging MD files in library/raw. The process tracks progress via a JSON file and resumes from the last completed file if interrupted.

Resume support for batch ingestion works by tracking per-file progress in a dedicated JSON file. If processing stops, it restarts from the last uncompleted file rather than starting over.

You can process markdown files concurrently during batch ingestion. It supports optional parallelism with up to five workers to speed up applying updates to staged files.

Batch ingestion does not modify existing files in the knowledge base library. It enforces consistency guarantees by ensuring existing library files remain untouched during the update process.

After batch ingesting staged markdown files completes, the process triggers a shelf-index rebuild and generates a consolidated log entry to finalize the knowledge base updates.

You need to stage new or updated MD files in the library/raw directory before starting batch knowledge base updates. These staged files serve as the input for the ingestion process.