skills.rest
majidraza1228
majidraza1228COMMUNITY
Explore all skills by this author →
WebsiteGitHub

evaluate-rag

Evaluate RAG pipelines by measuring retrieval and generation quality separately.

Updated Apr 19, 2026
Search Tags:#rag#metrics#evaluation#retrieval#nlp#generation#judges
One-click install
npx skills add https://github.com/majidraza1228/eval-framework --skill evaluate-rag-majidraza1228
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluate-rag
Source: https://github.com/majidraza1228/eval-framework/tree/main/skills/evaluate-rag
Command: npx skills add https://github.com/majidraza1228/eval-framework --skill evaluate-rag-majidraza1228

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Evaluate RAG pipelines by separately assessing retrieval quality and generated answer quality, enabling identification of bottlenecks and targeted improvements.

Core Features & Use Cases

  • Retrieval evaluation: measure Recall@k, MRR, and NDCG to verify the retriever returns relevant documents.
  • Generation evaluation: assess faithfulness, relevance, completeness, and format compliance using dedicated judges when necessary.
  • Bottleneck diagnosis: determine whether issues lie in retrieval or generation and guide fixes.
  • Real-world use: QA chat assistants that retrieve knowledge base documents to answer user queries.

Quick Start

Run the evaluation workflow on your RAG setup to obtain separate retrieval and generation scores.

Frequently Asked Questions about evaluate-rag

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG pipeline retrieval and generation quality separately?▼

To evaluate RAG pipelines, quantify retrieval quality using metrics like Recall@k, MRR, and NDCG, and assess generation quality through faithfulness and relevance judges to identify specific bottlenecks.

What metrics are used for retrieval evaluation in RAG systems?▼

Retrieval evaluation in RAG systems uses Recall@k, MRR, and NDCG to verify the retriever returns relevant documents. These metrics require labeled query-to-document mappings and access to the retriever to collect ranked results.

Do I need labeled data to evaluate RAG pipelines?▼

Yes, evaluating RAG pipelines requires labeled query-to-document mappings and access to the retriever to run queries and collect ranked results for calculating retrieval metrics and diagnosing generation issues.

How can I diagnose whether my RAG system bottleneck is retrieval or generation?▼

Diagnose RAG bottlenecks by separately scoring retrieval and generation. Measure retrieval metrics like Recall@k and NDCG, then use dedicated judges to assess generation faithfulness, relevance, completeness, and format compliance.

What generation evaluation criteria are assessed for RAG chat assistants?▼

Generation evaluation for RAG chat assistants assesses faithfulness, relevance, completeness, and format compliance using dedicated judges to ensure the generated answer quality meets real-world QA requirements.

Related Skills

Explore More Data & Analytics Skills→
floflo777floflo777
33

rag-eval

Evaluate RAG systems using retrieval, generation, and latency metrics.

Community
Intermediate
MantisWareMantisWare

eval-rag

Evaluate RAG pipelines with retrieval and generation metrics and bottleneck analysis.

Community
Advanced
Godhand-ArnavGodhand-Arnav

rag-architect

Design and implement end-to-end RAG pipelines for context-aware LLM responses.

Community
Advanced
Zhow01Zhow01
52

llm-evaluation

Assess and benchmark LLM outputs across automated metrics, human judgments, and safety checks.

Community
Advanced