skills.rest
Avi977
Avi977COMMUNITY
Explore all skills by this author →
GitHub

evaluate-rag

Evaluate RAG retrieval and generation with Recall@k, MRR, and NDCG@k metrics.

Updated Mar 17, 2026
Search Tags:#llm#rag#metrics#evaluation#retrieval#nlp#generation
One-click install
npx skills add https://github.com/Avi977/ace-claude-toolkit --skill evaluate-rag-avi977
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluate-rag
Source: https://github.com/Avi977/ace-claude-toolkit/tree/main/skills/evaluate-rag
Command: npx skills add https://github.com/Avi977/ace-claude-toolkit --skill evaluate-rag-avi977

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill helps teams quantify and optimize retrieval-augmented generation systems by providing a structured evaluation workflow for both the retrieval and generation components, enabling targeted improvements.

Core Features & Use Cases

  • Build retrieval evaluation datasets with queries and ground-truth chunks to measure Recall@k and other retrieval metrics.
  • Assess generation faithfulness and relevance by comparing outputs against retrieved context and applying error analysis.
  • Diagnose bottlenecks across retrieval, chunking, and prompting to guide systematic improvements in real-world RAG deployments.

Quick Start

Create a retrieval dataset and run the evaluation workflow to observe Recall@k, MRR, and NDCG@k.

Frequently Asked Questions about evaluate-rag

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG retrieval quality using metrics like Recall@k and NDCG@k?▼

Evaluate RAG retrieval quality by building a dataset with queries and ground-truth chunks, then computing Recall@k, Precision@k, MRR, and NDCG@k to measure ranking accuracy. This workflow provides reproducible dataset construction and clear metrics to diagnose retrieval bottlenecks systematically.

What metrics should I use to assess LLM generation faithfulness and relevance in a RAG system?▼

Assess LLM generation faithfulness and relevance by comparing LLM outputs against retrieved context and applying error analysis. This identifies whether generations are grounded in the context and relevant to the query, guiding improvements in prompting and chunking strategies.

How can I diagnose bottlenecks in my retrieval-augmented generation pipeline?▼

Diagnose RAG pipeline bottlenecks by evaluating both retrieval and generation components separately. Measure retrieval metrics like MRR and NDCG@k, analyze generation faithfulness, and apply the workflow's guidelines for chunking and safe prompting to target systematic improvements.

Do I need a specific framework or dependencies to measure RAG performance end-to-end?▼

No specific dependencies are required to measure RAG performance. The workflow provides structured prerequisites for reproducible dataset construction and evaluation guidelines, allowing you to quantify retrieval and generation quality using standard NLP metrics within your existing environment.

What is the best way to build a retrieval evaluation dataset for RAG?▼

Build a retrieval evaluation dataset by compiling queries paired with their corresponding ground-truth chunks. This reproducible dataset construction step is essential for accurately calculating retrieval metrics like Recall@k and Precision@k to measure and optimize pipeline performance.

When should I apply chunking guidelines to optimize my RAG system's retrieval?▼

Apply chunking guidelines when diagnosing retrieval bottlenecks to optimize your RAG system. Proper chunking directly impacts the accuracy of metrics like Recall@k and MRR by ensuring the retrieved context is optimally segmented for both retrieval precision and generation faithfulness.

Related Skills

Explore More Data & Analytics Skills→
Miosa-osaMiosa-osa
226

eval-rag

Evaluate RAG pipelines with retrieval and generation metrics plus bottleneck analysis.

Official
Advanced
marchattonmarchatton
5

evaluate-rag

Evaluate RAG pipelines with Recall@k, Precision@k, MRR, and NDCG@k metrics.

Community
Advanced
mupengi-botmupengi-bot
11

rag-engineer

Design RAG systems with embedding models, vector databases, and retrieval optimization.

Community
Advanced
ollieb89ollieb89

llm-evaluation

Evaluate LLM outputs with automated metrics and LLM-as-Judge workflows.

Community
Advanced