skills.rest
marchatton
marchattonCOMMUNITY
Explore all skills by this author →
Website𝕏@marchattonhereGitHub

evaluate-rag

Evaluate RAG pipelines with Recall@k, Precision@k, MRR, and NDCG@k metrics.

5|Updated Oct 22, 2025
Search Tags:#llm#rag#metrics#evaluation#retrieval#generation
One-click install
npx skills add https://github.com/marchatton/agent-skills --skill evaluate-rag
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluate-rag
Source: https://github.com/marchatton/agent-skills/tree/main/.agents/skills/08-evals/evaluate-rag
Command: npx skills add https://github.com/marchatton/agent-skills --skill evaluate-rag

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the critical need to systematically evaluate and improve the performance of Retrieval-Augmented Generation (RAG) systems by dissecting the quality of both retrieval and generation components.

Core Features & Use Cases

  • Component-wise Evaluation: Separates the assessment of retrieval accuracy from generation faithfulness and relevance.
  • Dataset Curation: Provides methods for generating manual and synthetic QA pairs for robust retrieval testing.
  • Metric Implementation: Details the application of key metrics like Recall@k, Precision@k, MRR, and NDCG@k for different query types.
  • Chunking Optimization: Guides the process of tuning chunking strategies for improved retrieval performance.
  • Use Case: When a RAG system is underperforming, use this Skill to pinpoint whether the issue lies in retrieving the correct documents or in the LLM's ability to synthesize a faithful and relevant answer from the provided context.

Quick Start

Use the evaluate-rag skill to analyze the retrieval quality of the RAG pipeline by generating synthetic QA pairs for a given document.

Frequently Asked Questions about evaluate-rag

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG pipeline retrieval quality and generation faithfulness?▼

Evaluating RAG pipeline performance requires separating retrieval accuracy from generation faithfulness. You assess retrieval using metrics like Recall@k and NDCG@k, while diagnosing generation failures by measuring context relevance and answer faithfulness.

What metrics should I use to measure retrieval accuracy in a RAG system?▼

Retrieval accuracy in a RAG system is measured using Recall@k, Precision@k, MRR, and NDCG@k. These metrics quantify how well the pipeline retrieves relevant documents for different query types.

How do I create test datasets for evaluating RAG systems?▼

Creating test datasets for evaluating RAG systems involves manual curation and synthetic QA generation. This process builds robust QA pairs from your documents to test retrieval pipeline accuracy.

How can I optimize chunking strategies to improve RAG retrieval performance?▼

Optimizing chunking strategies improves RAG retrieval performance by tuning document segmentation. This process adjusts how text is divided to maximize context relevance and ensure accurate document retrieval.

Why does my RAG system return inaccurate or unfaithful answers?▼

RAG systems return inaccurate answers due to retrieval failures or generation unfaithfulness. Diagnose whether the pipeline retrieves incorrect context or if the LLM fails to synthesize a relevant answer from the provided context.

Can I diagnose LLM generation failures based on context relevance in a RAG pipeline?▼

Diagnosing LLM generation failures based on context relevance is possible by evaluating RAG pipelines. This process isolates whether the issue stems from retrieving incorrect documents or the LLM synthesizing unfaithful answers.

Related Skills

Explore More Software Engineering Skills→
ricablericable
1

ruvector-agentic-synth

Generates synthetic data for AI/ML training, RAG evaluation, and agentic workflow testing.

Community
Intermediate
spideynolovespideynolove
2

embedding-strategies

Compare embedding models and define chunking strategies for RAG pipelines.

Community
Advanced
floflo777floflo777
33

rag-eval

Evaluate RAG systems using retrieval, generation, and latency metrics.

Community
Intermediate
DrLuggelsDrLuggels
3

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Community
Advanced