skills.rest
kinnerkarmanish
kinnerkarmanishCOMMUNITY
Explore all skills by this author →
GitHub

eval-rag

Evaluate RAG pipelines by scoring retrieval and generation against gold references.

Updated Apr 26, 2026
Search Tags:#rag#benchmarking#metrics#evaluation#retrieval#generation
One-click install
npx skills add https://github.com/kinnerkarmanish/mak --skill eval-rag-kinnerkarmanish
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-rag
Source: https://github.com/kinnerkarmanish/mak/tree/main/library/skills/ai-patterns/eval-rag
Command: npx skills add https://github.com/kinnerkarmanish/mak --skill eval-rag-kinnerkarmanish

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluates end-to-end RAG pipelines by separately scoring retrieval and generation, then correlates results to identify bottlenecks and improvement opportunities.

Core Features & Use Cases

  • Retrieval metrics: Recall@K, Precision@K, MRR, NDCG@K, and Hit Rate to quantify how well the system fetches relevant documents.
  • Generation metrics: Faithfulness, Relevance, Completeness, and Conciseness to assess answer quality against retrieved context.
  • Bottleneck analysis: Cross-reference retrieval and generation scores to classify failures and guide optimization.
  • Pipeline comparison: Side-by-side evaluation of multiple configurations with statistical significance testing.
  • Use case: Benchmarking RAG setups in knowledge bases, customer-support bots, or any system that relies on retrieved content to answer queries.

Quick Start

Run the eval-rag tool with your pipeline configuration, evaluation queries, and golden answers to generate a full evaluation report.

Frequently Asked Questions about eval-rag

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate end-to-end RAG quality for my knowledge base QA system?▼

To evaluate RAG quality, you need a labeled evaluation set with queries and golden answers, plus your pipeline configurations. The tool measures retrieval and generation performance separately against gold references, producing per-query metrics and an overall report to identify bottlenecks.

What metrics are used for benchmarking RAG retrieval and generation performance?▼

RAG benchmarking uses retrieval metrics like Recall@K, Precision@K, MRR, NDCG@K, and Hit Rate to quantify document fetching. Generation performance is measured using Faithfulness, Relevance, Completeness, and Conciseness to assess answer quality against retrieved context.

How can I find bottlenecks in my RAG pipeline using retrieval and generation scores?▼

Bottleneck analysis cross-references retrieval and generation scores to classify failures and guide optimization. By separately scoring how well the system fetches relevant documents and assesses answer quality, you can pinpoint whether issues originate in retrieval or generation.

Can I compare multiple RAG pipeline configurations with statistical significance testing?▼

Yes, you can perform side-by-side evaluation of multiple RAG configurations with statistical significance testing. This allows you to benchmark different RAG setups, such as those used in customer-support bots or content search systems, to identify the best performing pipeline.

Do I need golden answers to measure RAG quality for customer support bots?▼

Yes, measuring RAG quality requires a labeled evaluation set containing queries and golden answers. Along with your pipeline configurations, these gold references are necessary to accurately score how well the system fetches documents and generates relevant responses.

Why does my RAG system retrieve relevant documents but still generate poor answers?▼

This indicates a generation bottleneck. By cross-referencing retrieval and generation scores, you can classify failures and guide optimization, determining if issues stem from document fetching or the generation phase's faithfulness, relevance, completeness, or conciseness.

Related Skills

Explore More Data & Analytics Skills→
Miosa-osaMiosa-osa
226

eval-rag

Evaluate RAG pipelines with retrieval and generation metrics plus bottleneck analysis.

Official
Advanced
MantisWareMantisWare

eval-rag

Evaluate RAG pipelines with retrieval and generation metrics and bottleneck analysis.

Community
Advanced
zhangzhang-111-izhangzhang-111-i
2

rag-architect

Designs RAG pipelines with chunking, embedding, retrieval, and evaluation strategies.

Community
Advanced
roanbrasilroanbrasil
5

llm-evals

Implement LLM evaluation patterns with LangSmith, RAGAS, and PromptFoo.

Community
Advanced