rag-ingestion-pipeline

Ingest MD, HTML, and PDF documents into RAG pipelines with hashing and chunking.

9|2|Updated Nov 28, 2025

Search Tags:#RAG #ingestion #chunking #embedding #vector-db

Ready to run

npx skills add https://github.com/mjunaidca/robolearn --skill rag-ingestion-pipeline

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: rag-ingestion-pipeline
Source: https://github.com/mjunaidca/robolearn/tree/main/.claude/skills/engineering/rag-ingestion-pipeline
Command: npx skills add https://github.com/mjunaidca/robolearn --skill rag-ingestion-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Provides end-to-end ingestion for documents into a RAG system with chunking, embedding, and vector upserts.

Core Features & Use Cases

Crawler to discover docs
Frontmatter parsing and front matter hashing for change detection
Chunking with overlap and embedding
Upsert to vector store with payload indexing

Quick Start

Run the ingestion pipeline to ingest a chapter’s docs and build a retrievable index.

Frequently Asked Questions about rag-ingestion-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I set up an RAG ingestion pipeline for documents?▼

An RAG ingestion pipeline processes documents through stages: crawling, parsing frontmatter, semantic chunking into 400–512 token chunks with overlap, embedding, and upserting to a vector store. This Skill implements all stages with content hashing for incremental updates, handling MD, HTML, and PDF formats end to end.

What's the best way to chunk documents for RAG systems?▼

Semantic chunking splits documents at meaningful boundaries rather than fixed sizes, targeting 400–512 tokens per chunk with 10–20% overlap to preserve context. This approach maintains semantic relationships and improves retrieval quality compared to naive splitting.

How does content hashing enable incremental document updates in RAG?▼

Content hashing with SHA-256 detects which documents have changed since the last ingestion. This allows the pipeline to skip processing unchanged content, reducing embedding costs and vector store operations while keeping your index current.

Can I use this pipeline with multiple document formats?▼

Yes, this pipeline supports MD, HTML, and PDF documents. It extracts frontmatter, applies semantic chunking, and handles format-specific parsing, enabling a unified ingestion workflow across document types.

Does this ingestion pipeline support multitenant vector stores?▼

Yes, the pipeline implements payload indexing for multitenant vector stores, allowing you to ingest and retrieve documents while maintaining tenant isolation through indexed metadata.

What happens to chunk relationships during ingestion?▼

The pipeline tracks chunk relationships during semantic chunking and embedding, preserving context links between adjacent chunks. This enables downstream retrieval to reconstruct document flow and improve answer quality.