text-embeddings-rag

What problem does it solve?

Turn unstructured document collections into dense embeddings and reliable retrieval context so that language models can answer questions accurately without brittle keyword searches or manual document inspection. The skill consolidates guidance on embedding generation, vector indexing, retrieval tuning, and integration with LLM inference to make RAG pipelines reproducible and maintainable.

Core Features & Use Cases

• Local embedding generation using sentence-transformers with GPU-accelerated batch encoding and L2 normalization for cosine search.
• Vector store integration patterns and validation for FAISS, ChromaDB, and Qdrant including persistence, ID mapping, and payload/metadata handling.
• End-to-end RAG pipelines: document chunking, embedding, indexing, retrieval, and generation with examples for retrieval + re-ranking and hybrid BM25+vector search.
• Retrieval optimization techniques: chunk size tuning, overlap strategies, cross-encoder re-ranking, reciprocal rank fusion, and metadata filtering.
• Language-specific guidance such as Vietnamese VN-MTEB recommendations and word-segmentation advice for BM25 hybrid search.
• Use Case: Index internal product docs, retrieve top context for a user question, and synthesize a grounded answer with source attributions.

Quick Start

Use the text-embeddings-rag skill to index your document corpus with sentence-transformers, persist vectors in FAISS or ChromaDB, and run a retrieval-augmented query to generate a context-aware answer.