skills.rest
ayeshakhalid192007-dev
ayeshakhalid192007-devCOMMUNITY
Explore all skills by this author →
GitHub

RAG Ingestion Pipeline Skill

Ingest Markdown, HTML, and PDF documents into Qdrant with semantic chunking.

1|Updated Feb 19, 2026
Search Tags:#embeddings#rag#ingestion#qdrant#chunking#semantic-search#incremental-update
One-click install
npx skills add https://github.com/ayeshakhalid192007-dev/humanoid-ai-studio --skill rag-ingestion-pipeline-skill
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: RAG Ingestion Pipeline Skill
Source: https://github.com/ayeshakhalid192007-dev/humanoid-ai-studio/tree/main/.claude/skills/rag-ingestion-pipeline
Command: npx skills add https://github.com/ayeshakhalid192007-dev/humanoid-ai-studio --skill rag-ingestion-pipeline-skill

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill removes the complexity of turning large documentation sets (Markdown, HTML, PDF) into a production-ready vector index for retrieval-augmented generation, enabling incremental updates and avoiding full re-indexing.

Core Features & Use Cases

  • Incremental Change Detection: Use content hashing to detect new, modified, or deleted files and only re-process changed content.
  • Semantic Chunking: Split by semantic boundaries (headers), target token sizes (400–512 tokens) and apply 10–20% overlap to preserve context.
  • Batched Embeddings & Vector Uploads: Efficiently embed chunks in batches and upsert them into Qdrant with indexed payloads for filtered retrieval.
  • Production Patterns: Includes crawler, frontmatter parser, deterministic chunk IDs, Qdrant-native state tracking, and background ingestion APIs with webhook triggers.
  • Use Case: Ideal for course books or technical documentation where modules update frequently and filtered, contextual retrieval is required.

Quick Start

Trigger an incremental ingestion job for the repo docs path to detect changes, chunk semantically, embed in batches, and upsert into a Qdrant collection.

Frequently Asked Questions about RAG Ingestion Pipeline Skill

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I incrementally update a Qdrant vector index for technical documentation?▼

To incrementally update a Qdrant vector index, use content hashing to detect new, modified, or deleted files, then semantically chunk and embed only the changed content before upserting into the collection.

What is the best way to chunk PDF and HTML documents for semantic search?▼

The best way to chunk PDF and HTML documents for semantic search is splitting by header boundaries, targeting 400–512 token sizes, and applying 10–20% overlap to preserve context for retrieval.

Does Qdrant support filtered retrieval for RAG ingestion pipelines?▼

Yes, Qdrant supports filtered retrieval for RAG ingestion pipelines by indexing chunk payloads during the batched embedding and upsert process, enabling contextual and filtered queries.

How do I avoid full re-indexing when updating multi-module course books?▼

Avoid full re-indexing of multi-module course books by applying incremental change detection via content hashing, which isolates modified files and only re-processes the updated content.

Can I trigger background ingestion for Markdown files using webhook APIs?▼

Yes, you can trigger background ingestion for Markdown files using webhook APIs, which initiate crawling and frontmatter parsing to detect changes and embed content in batches.

Related Skills

Explore More Software Engineering Skills→
mjunaidcamjunaidca
9

rag-ingestion-pipeline

Ingest MD, HTML, and PDF documents into RAG pipelines with hashing and chunking.

Community
Advanced
ivanshamaevivanshamaev
14

rag-data-pipeline

Build RAG ingestion and query pipelines with hybrid retrieval and reranking.

Community
Advanced
ricardorochericardoroche
2

rag-design-patterns

Standardizes chunking, vector database integration, embedding management, and retrieval reranking for RAG pipelines.

Community
Intermediate
christhz666christhz666

defuddle

Extract readable markdown content from web pages via Defuddle CLI.

Community
Basic