skills.rest
rocklambros
rocklambrosCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

evaluating-rag-retrieval

Evaluate RAG pipelines with retrieval metrics, generation scores, and failure attribution.

2|Updated May 23, 2026
Search Tags:#rag#faithfulness#mrr#ndcg#retrieval evaluation#bootstrap ci#failure attribution
One-click install
npx skills add https://github.com/rocklambros/rcs --skill evaluating-rag-retrieval
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluating-rag-retrieval
Source: https://github.com/rocklambros/rcs/tree/main/skills/ml-datasci/evaluating-rag-retrieval
Command: npx skills add https://github.com/rocklambros/rcs --skill evaluating-rag-retrieval

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill evaluates Retrieval-Augmented Generation systems without collapsing retrieval and generation into a single misleading score. It helps teams pinpoint whether poor answers come from search quality, chunking, reranking, prompt grounding, or the model’s use of retrieved context.

Core Features & Use Cases

  • Stage-separated metrics: Reports retrieval metrics such as recall@k, MRR, and nDCG alongside generation metrics such as faithfulness, answer relevance, and context utilization.
  • Failure attribution: Builds a retrieval-hit versus retrieval-miss and generation-correct versus generation-wrong matrix to show exactly where the pipeline fails.
  • Evaluation workflow: Validates golden-set quality, checks for leakage, uses bootstrap confidence intervals, and recommends the next experiment for RAG regression testing, embedding-model changes, chunking changes, or reranker comparisons.

Quick Start

Ask the skill to evaluate your RAG pipeline end to end using a golden question-answer set, then report retrieval metrics, generation metrics on retrieval hits, failure attribution, and the highest-leverage next experiment.

Frequently Asked Questions about evaluating-rag-retrieval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG retrieval quality separately from generation quality?▼

To evaluate RAG retrieval quality separately, you can report retrieval metrics like recall@k, MRR, and nDCG alongside generation metrics like faithfulness and answer relevance to isolate pipeline failures.

What is failure attribution in a RAG pipeline?▼

Failure attribution in a RAG pipeline uses a 2x2 matrix of retrieval hits versus misses and generation correctness to pinpoint whether poor answers come from search, chunking, reranking, or prompt grounding.

Why does my RAG system feel worse than a direct LLM?▼

Your RAG system might feel worse than a direct LLM if retrieval misses prevent grounding or if the model fails to use context, which a failure-attribution analysis can diagnose stage by stage.

How do I compare rerankers or chunking strategies for retrieval evaluation?▼

To compare rerankers or chunking changes, you can use a golden question-answer set to calculate retrieval metrics, generation judge scoring, and bootstrap confidence intervals for reliable regression testing.

How do I validate a golden set for RAG evaluation?▼

To validate a golden set for RAG evaluation, you must check for data leakage, calculate stage-separated metrics, and apply bootstrap confidence intervals to ensure your baseline scoring is robust.

What metrics do I need for RAG retrieval evaluation?▼

RAG retrieval evaluation requires retrieval metrics like recall@k, MRR, and nDCG, plus generation judge scoring for faithfulness and context utilization to build a complete failure-attribution matrix.

Related Skills

Explore More Data & Analytics Skills→
sayalinvidiasayalinvidia

rag-eval

Run filesystem-based RAG benchmarks and output NVSkills metrics.

Community
Advanced
agentsopeagentsope
287

agentsop-regression-gate

Enforce LLM regression testing in CI with a held-out eval set and noise-aware threshold.

Community
Advanced
floflo777floflo777
33

rag-eval

Evaluate RAG systems using retrieval, generation, and latency metrics.

Community
Intermediate
pluginagentmarketplacepluginagentmarketplace
2

evaluation-metrics

Evaluate LLM outputs using BLEU, ROUGE, BERTScore, and RAG metrics.

Community
Advanced