rag-data-pipeline

What problem does it solve?

This Skill helps you design and debug end-to-end Retrieval-Augmented Generation (RAG) data pipelines so retrieval quality stays high as your knowledge base grows and changes.

Core Features & Use Cases

• Document ingestion & chunking: Choose fixed-size, recursive, semantic, or structure-aware chunking aligned to your document types and constraints.
• Embedding & vector storage: Generate embeddings with OpenAI/Cohere/local sentence-transformers and upsert them into pgvector, Chroma, Qdrant, or Weaviate with metadata.
• Hybrid retrieval & re-ranking: Combine dense + BM25 retrieval using RRF, then optionally apply cross-encoder re-ranking and metadata filters.
• Incremental refresh & monitoring: Refresh only changed chunks (hash-based), orchestrate ingestion with Airflow, and track hit rate, MRR, and latency.

Quick Start

Use the rag-data-pipeline skill to build an indexing pipeline that chunks your PDFs into structure-aware sections, embeds them, stores them in a vector database, and answers user queries using hybrid retrieval with optional cross-encoder re-ranking.