knowledge-persistence

Chunk documents, generate QA pairs, and persist to Milvus.

1|Updated Apr 12, 2026

Search Tags:#retrieval #chunking #milvus #knowledge-persistence #qa-generation #doc2query

Ready to run

npx skills add https://github.com/mengsi16/brain-base --skill knowledge-persistence-mengsi16

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: knowledge-persistence
Source: https://github.com/mengsi16/brain-base/tree/main/skills/knowledge-persistence
Command: npx skills add https://github.com/mengsi16/brain-base --skill knowledge-persistence-mengsi16

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

当 get-info-agent 或 upload-agent 已拿到清洗/转换后的文档草稿，需要把知识工业级写入本地和检索层时触发。负责 LLM 分块、5000 字符阈值约束、合成 QA 问题生成、raw/chunks 双落盘、Milvus hybrid 持久化，以及 SQLite 关键词更新。

Core Features & Use Cases

生成/接收 raw Markdown：将清洗后的草稿以 raw Markdown 保存并准备进入分块流程。
语义分块与 QA 生成：使用 LLM 将文档分块（受 5000 字符硬阈值约束），并为每个 chunk 生成 3〜5 条 doc2query 问题，写回 chunk frontmatter 的 questions 字段。
持久化与更新：把 chunks 和相关问题写入 raw/chunks、执行 Milvus hybrid 入库，以及更新 keywords.db 与 priority.json。

Quick Start

触发知识持久化流程，将一个清洗后的文档草稿提交给本 skill，进行分块、QA 生成与持久化。

Frequently Asked Questions about knowledge-persistence

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I persist documents to Milvus and generate QA pairs for retrieval?▼

To persist documents to Milvus, you can chunk documents with a 5000-character threshold, generate 3-5 doc2query QA pairs per chunk, and ingest them into Milvus hybrid retrieval alongside local storage.

What is the character limit for document chunking when preparing files for vector retrieval?▼

The character limit for document chunking is a hard threshold of 5000 characters per block. This ensures chunks remain optimal for generating QA pairs and ingesting into Milvus hybrid retrieval.

How do I generate synthetic questions from text chunks to improve search retrieval?▼

You can generate synthetic questions from text chunks using the doc2query mechanism, which creates 3 to 5 QA pairs per chunk and writes them into the chunk frontmatter's questions field to enhance retrieval.

How to save document chunks locally while updating keywords and priority metadata?▼

You can save document chunks locally by writing raw markdown and chunks to disk, then updating the SQLite keywords database and priority.json file to manage metadata alongside Milvus hybrid ingestion.

Does this chunking and QA generation workflow require pre-cleaned markdown documents?▼

Yes, the workflow requires pre-cleaned markdown documents arriving from upstream processes. It handles raw markdown storage and processes it through chunking, QA generation, and Milvus persistence.

What is the best way to build an end-to-end knowledge persistence pipeline using Milvus?▼

The best way to build a knowledge persistence pipeline is using an automated workflow that handles LLM chunking, doc2query QA generation, dual local disk storage, and Milvus hybrid ingestion with keyword management.