knowledge-persistence

To persist documents to Milvus, you can chunk documents with a 5000-character threshold, generate 3-5 doc2query QA pairs per chunk, and ingest them into Milvus hybrid retrieval alongside local storage.

The character limit for document chunking is a hard threshold of 5000 characters per block. This ensures chunks remain optimal for generating QA pairs and ingesting into Milvus hybrid retrieval.

You can generate synthetic questions from text chunks using the doc2query mechanism, which creates 3 to 5 QA pairs per chunk and writes them into the chunk frontmatter's questions field to enhance retrieval.

You can save document chunks locally by writing raw markdown and chunks to disk, then updating the SQLite keywords database and priority.json file to manage metadata alongside Milvus hybrid ingestion.

Yes, the workflow requires pre-cleaned markdown documents arriving from upstream processes. It handles raw markdown storage and processes it through chunking, QA generation, and Milvus persistence.

The best way to build a knowledge persistence pipeline is using an automated workflow that handles LLM chunking, doc2query QA generation, dual local disk storage, and Milvus hybrid ingestion with keyword management.