skills.rest
ahuaracab
ahuaracabCOMMUNITY
Explore all skills by this author →
GitHub

rag-testing

Validates RAG pipelines across chunking, embeddings, retrieval, ranking, and answer generation layers.

Updated Aug 31, 2026
Search Tags:#semantic search#embeddings#rag#evaluation#retrieval#hallucination#bm25
One-click install
npx skills add https://github.com/ahuaracab/test-genesis --skill rag-testing-ahuaracab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-testing
Source: https://github.com/ahuaracab/test-genesis/tree/main/.agents/skills/rag-testing
Command: npx skills add https://github.com/ahuaracab/test-genesis --skill rag-testing-ahuaracab

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Retrieval-Augmented Generation systems fail in two distinct ways: retrieval failures (wrong documents fetched) and context-answer mismatches (right documents, wrong answer). Without layer-by-layer evaluation, teams cannot tell whether to fix chunking, embeddings, retrieval ranking, or the LLM prompt when answers degrade. ## Core Features & Use Cases - Five-Layer Validation: Independently scores chunking coherence, embedding quality, retrieval precision/recall/MRR, hybrid ranking via Reciprocal Rank Fusion, and end-to-end answer correctness with hallucination detection. - Hybrid Search Comparison: Benchmarks semantic-only retrieval against semantic + BM25 fusion to catch exact-match queries like discount codes or incident IDs. - Production Gates: Multi-run evaluation with confidence intervals per IEEE 29119-2, plus CI/CD thresholds such as retrieval precision ≥ 0.85 and hallucination rate ≤ 0.05. - Use Case: A chatbot answering questions from a policy knowledge base retrieves the refund policy but hallucinates processing times. The skill isolates the failure to Layer 5 (generation) rather than Layer 3 (retrieval), so the team fixes the prompt instead of re-chunking documents. ## Quick Start Ask the AI to evaluate your RAG pipeline by running the 5-layer assessment against your knowledge base chunks and a set of golden test queries with known relevant documents.

Frequently Asked Questions about rag-testing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I test a RAG pipeline end to end?▼

Test each of the five layers independently first: chunking coherence, embedding similarity, retrieval precision and recall, hybrid ranking, and answer correctness. Then run the full pipeline with golden queries where you know the correct documents and expected answers.

What metrics measure retrieval quality in RAG systems?▼

Use precision@k to measure how many top results are relevant, recall@k to measure how many relevant documents were found, and mean reciprocal rank to check how early the first relevant document appears. Production targets are typically recall above 80% and MRR above 0.7.

Semantic search vs hybrid search for RAG retrieval?▼

Hybrid search combining semantic embeddings with BM25 lexical matching via Reciprocal Rank Fusion outperforms semantic-only retrieval. Semantic search misses exact-match queries like product codes, while BM25 misses conceptual queries, so fusing both covers both failure modes.

Which embedding model should I use for RAG?▼

VoyageAI offers the best quality-to-cost balance for production, OpenAI text-embedding-3 is a mature alternative, and Ollama models like nomic-embed-text work free locally with lower quality. Validate your choice by testing cosine similarity on known similar and unrelated text pairs.

Why does my RAG system hallucinate answers?▼

Hallucination occurs when the LLM generates facts not present in the retrieved documents. Detect it by comparing answer terms against retrieved context, and separate this Layer 5 failure from retrieval failures so you fix the prompt or model rather than the search index.

When should I not use RAG testing?▼

Do not use it for evaluating standalone LLM prompts without retrieval, since prompt-evaluation covers that case. It also does not handle embedding hyperparameter tuning or load testing retrieval latency at scale, which require separate profiling tools.

Related Skills

Explore More Software Engineering Skills→
langwatchlangwatch
3.5k

generate-rag-dataset

Generate synthetic Q&A evaluation datasets from a RAG knowledge base for LangWatch experiments.

Official
Intermediate
oalansilvaoalansilva

performance-budgeting

Define and enforce performance budgets across CI/CD pipelines.

Community
Advanced
applied-artificial-intelligenceapplied-artificial-intelligence
81

llm-evaluation

Evaluate LLM applications with automated metrics and LLM-as-judge scoring.

Official
Advanced
Netflix-SkunkworksNetflix-Skunkworks
76

railguard-benchmark

Store security scan findings in SQLite and compare runs against answer sheets.

Official
Advanced