agentsop-multiscale-chunking

What problem does it solve?

This Skill fixes retrieval-augmented generation failures where a single chunk size cannot simultaneously achieve high retrieval precision and sufficient surrounding context, causing either fragmentary answers or diluted relevance on long documents.

Core Features & Use Cases

• Chunk-paradox resolution (C5 overlay): decouples the embed unit from the return unit by embedding small for matching while returning a larger context unit for synthesis.
• Three-gate SOP: runs a chunk-size sweep first, then selects a horizontal (sentence-window) vs vertical (auto-merging) expansion strategy based on document structure, and finally measures the lift or reverts.
• Operation models and dilemma cases: provides MSC-01…MSC-07 steps, plus decision logic for Sentence-Window vs Auto-Merging and for when the chunk-size sweep does not converge.
• Cross-framework mapping for implementation clarity: aligns LlamaIndex concepts (SentenceWindowNodeParser, MetadataReplacementPostProcessor, HierarchicalNodeParser, AutoMergingRetriever) with the LangChain ParentDocumentRetriever analogue.

Quick Start

Ask an AI engineer to apply the agentsop-multiscale-chunking overlay to your RAG ingestion and retrieval codebase after your chunk-size sweep shows a non-flat frontier, so retrieval precision and answer context improve together.