evaluating-rag-retrieval

What problem does it solve?

This Skill evaluates Retrieval-Augmented Generation systems without collapsing retrieval and generation into a single misleading score. It helps teams pinpoint whether poor answers come from search quality, chunking, reranking, prompt grounding, or the model’s use of retrieved context.

Core Features & Use Cases

• Stage-separated metrics: Reports retrieval metrics such as recall@k, MRR, and nDCG alongside generation metrics such as faithfulness, answer relevance, and context utilization.
• Failure attribution: Builds a retrieval-hit versus retrieval-miss and generation-correct versus generation-wrong matrix to show exactly where the pipeline fails.
• Evaluation workflow: Validates golden-set quality, checks for leakage, uses bootstrap confidence intervals, and recommends the next experiment for RAG regression testing, embedding-model changes, chunking changes, or reranker comparisons.

Quick Start

Ask the skill to evaluate your RAG pipeline end to end using a golden question-answer set, then report retrieval metrics, generation metrics on retrieval hits, failure attribution, and the highest-leverage next experiment.