rag-ingestion-pipeline

An RAG ingestion pipeline processes documents through stages: crawling, parsing frontmatter, semantic chunking into 400–512 token chunks with overlap, embedding, and upserting to a vector store. This Skill implements all stages with content hashing for incremental updates, handling MD, HTML, and PDF formats end to end.

Semantic chunking splits documents at meaningful boundaries rather than fixed sizes, targeting 400–512 tokens per chunk with 10–20% overlap to preserve context. This approach maintains semantic relationships and improves retrieval quality compared to naive splitting.

Content hashing with SHA-256 detects which documents have changed since the last ingestion. This allows the pipeline to skip processing unchanged content, reducing embedding costs and vector store operations while keeping your index current.

Yes, this pipeline supports MD, HTML, and PDF documents. It extracts frontmatter, applies semantic chunking, and handles format-specific parsing, enabling a unified ingestion workflow across document types.

Yes, the pipeline implements payload indexing for multitenant vector stores, allowing you to ingest and retrieve documents while maintaining tenant isolation through indexed metadata.

The pipeline tracks chunk relationships during semantic chunking and embedding, preserving context links between adjacent chunks. This enables downstream retrieval to reconstruct document flow and improve answer quality.