skills.rest
hamelsmu
hamelsmuCOMMUNITY
Explore all skills by this author →
Website𝕏@HamelHusainGitHub

evaluate-rag

Evaluate RAG pipelines by measuring retrieval metrics and generation quality.

1.6k|164|Updated Mar 1, 2026
Search Tags:#llm#rag#evaluation#retrieval#nlp#generation
One-click install
npx skills add https://github.com/hamelsmu/evals-skills --skill evaluate-rag-hamelsmu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluate-rag
Source: https://github.com/hamelsmu/evals-skills/tree/main/skills/evaluate-rag
Command: npx skills add https://github.com/hamelsmu/evals-skills --skill evaluate-rag-hamelsmu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill addresses the challenge of evaluating the performance of Retrieval-Augmented Generation (RAG) systems by providing a structured approach to assess both retrieval and generation components.

Core Features & Use Cases

  • Component-wise Evaluation: Separates the assessment of retrieval quality (e.g., Recall@k, MRR) from generation quality (faithfulness, relevance).
  • Dataset Generation: Offers methods for creating evaluation datasets, including manual curation and synthetic QA pair generation.
  • Chunking Optimization: Guides users on tuning chunking strategies (size, overlap, content-awareness) to improve retrieval.
  • Use Case: Debugging a RAG system that returns irrelevant information or generates factually incorrect answers by pinpointing whether the issue lies in document retrieval or the LLM's response generation.

Quick Start

Use the evaluate-rag skill to assess the retrieval quality of the RAG pipeline using the Recall@5 metric.

Frequently Asked Questions about evaluate-rag

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG pipeline retrieval and generation quality?▼

To evaluate RAG pipeline quality, this Skill separates retrieval assessment using metrics like Recall@k, MRR, and NDCG@k from generation assessment, measuring faithfulness and relevance to pinpoint specific component failures.

Why does my RAG system return irrelevant information and factually incorrect answers?▼

RAG systems return irrelevant information or factually incorrect answers due to failures in either document retrieval or LLM generation. This Skill diagnoses the root cause by analyzing metric patterns across both components.

What is the best way to build a retrieval evaluation dataset for RAG?▼

Building a retrieval evaluation dataset for RAG can be achieved through manual curation or synthetic QA pair generation, providing a baseline to accurately measure retrieval metrics like Recall@k and MRR.

How do I optimize chunking strategies to improve RAG retrieval?▼

Optimize chunking strategies to improve RAG retrieval by tuning chunk size, overlap, and content-awareness. This Skill guides adjustments to these parameters to enhance the accuracy of document retrieval.

Can I assess LLM generation faithfulness and relevance separately from retrieval?▼

Yes, you can assess LLM generation faithfulness and relevance separately from retrieval. This Skill performs component-wise evaluation to isolate generation quality from document retrieval effectiveness.

Related Skills

Explore More Software Engineering Skills→
marchattonmarchatton
5

evaluate-rag

Evaluate RAG pipelines with Recall@k, Precision@k, MRR, and NDCG@k metrics.

Community
Advanced
latestaiagentslatestaiagents
5

chunking-strategies

Guide document chunking strategies for RAG systems with code examples.

Official
Intermediate
imMamdouhaboammarimMamdouhaboammar

agency-rag-pipeline-engineer

Design and implement retrieval-augmented generation pipelines with vector databases and LLM frameworks.

Community
Advanced
JhabbigJhabbig

llm-evaluation

Design evaluation frameworks for LLM applications with automated and human scoring.

Community
Advanced