skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

rag-data-pipeline

Build RAG ingestion and query pipelines with hybrid retrieval and reranking.

14|1|Updated May 5, 2026
Search Tags:#embeddings#rag#vector search#airflow#chunking#reranking#hybrid retrieval
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill rag-data-pipeline
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-data-pipeline
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/rag_pipeline
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill rag-data-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you design and debug end-to-end Retrieval-Augmented Generation (RAG) data pipelines so retrieval quality stays high as your knowledge base grows and changes.

Core Features & Use Cases

  • Document ingestion & chunking: Choose fixed-size, recursive, semantic, or structure-aware chunking aligned to your document types and constraints.
  • Embedding & vector storage: Generate embeddings with OpenAI/Cohere/local sentence-transformers and upsert them into pgvector, Chroma, Qdrant, or Weaviate with metadata.
  • Hybrid retrieval & re-ranking: Combine dense + BM25 retrieval using RRF, then optionally apply cross-encoder re-ranking and metadata filters.
  • Incremental refresh & monitoring: Refresh only changed chunks (hash-based), orchestrate ingestion with Airflow, and track hit rate, MRR, and latency.

Quick Start

Use the rag-data-pipeline skill to build an indexing pipeline that chunks your PDFs into structure-aware sections, embeds them, stores them in a vector database, and answers user queries using hybrid retrieval with optional cross-encoder re-ranking.

Frequently Asked Questions about rag-data-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a RAG data pipeline for document ingestion and vector search?▼

To build a RAG data pipeline, you chunk raw documents into structure-aware sections, generate embeddings, upsert them into a vector store like Qdrant or pgvector, and query using hybrid retrieval to ground LLM responses.

What is the best way to combine dense and BM25 retrieval using reciprocal ranked fusion?▼

Combining dense and BM25 retrieval using reciprocal ranked fusion (RRF) merges sparse and dense search results, improving candidate generation recall before optionally applying cross-encoder reranking for higher accuracy.

How do I implement incremental refresh and change detection for vector store upserts?▼

Incremental refresh for vector store upserts uses content hashes to deduplicate chunks and reproducible chunk identifiers, ensuring only changed text is re-embedded and updated in the vector database.

How do I monitor retrieval quality metrics like hit rate and MRR for RAG pipelines?▼

Monitoring retrieval quality metrics like hit rate, MRR, and latency tracks the effectiveness of your RAG pipeline as your knowledge base grows, ensuring query response relevance and speed remain high.

Does Airflow work with RAG data pipelines for orchestrating document ingestion?▼

Yes, Airflow orchestrates RAG data pipeline document ingestion, scheduling the chunking, embedding, and vector store upsert processes to maintain up-to-date indices for retrieval-augmented generation.

What chunking strategy should I use for structure-aware PDF sections in a RAG pipeline?▼

For structure-aware PDF sections in a RAG pipeline, you can choose fixed-size, recursive, semantic, or structure-aware chunking strategies aligned to your specific document types and retrieval constraints.

Related Skills

Explore More Software Engineering Skills→
jeffvincentjeffvincent
5

UXR Report Analyzer

Extract UXR report PDFs into standardized Markdown analyses with metadata.

Community
Advanced
Mathews-TomMathews-Tom
310

rag-auditor

Evaluate RAG pipeline retrieval and generation quality with metrics.

Community
Advanced
DrLuggelsDrLuggels
3

similarity-search-patterns

Provide patterns for similarity search with vector databases like Pinecone and Qdrant.

Community
Advanced
Kingly-AgencyKingly-Agency

rag-implementation

Build RAG pipelines that ground LLM responses with retrieved external knowledge.

Official
Advanced