eval-rag

Name: eval-rag
Availability: InStock
Author: Miosa

Evaluate RAG pipelines with retrieval and generation metrics plus bottleneck analysis.

226|55|Updated Mar 19, 2026

Search Tags:#rag #metrics #evaluation #retrieval #bottleneck #generation

Ready to run

npx skills add https://github.com/Miosa-osa/canopy --skill eval-rag-miosa-osa

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: eval-rag
Source: https://github.com/Miosa-osa/canopy/tree/main/library/skills/ai-patterns/eval-rag
Command: npx skills add https://github.com/Miosa-osa/canopy --skill eval-rag-miosa-osa

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Assess retrieval and generation quality in Retrieval-Augmented Generation pipelines by measuring retrieval effectiveness (recall, precision, MRR, NDCG) and generation fidelity (faithfulness, relevance, completeness), then surfacing bottlenecks for targeted improvements.

Core Features & Use Cases

End-to-end evaluation across retrieval and generation to quantify performance
Bottleneck diagnosis to distinguish retrieval issues from generation issues
Support for comparing multiple pipelines using golden answers and queries

Quick Start

Run /eval-rag with your pipeline configuration, queries, and golden answers to generate a comprehensive evaluation report.

Frequently Asked Questions about eval-rag

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I evaluate RAG pipeline retrieval and generation quality?▼

To evaluate RAG pipeline quality, you measure retrieval effectiveness using metrics like recall, precision, MRR, and NDCG, alongside generation fidelity metrics for faithfulness, relevance, and completeness. This end-to-end evaluation requires a configured pipeline, test queries, and golden answers.

What is the best way to diagnose bottlenecks in a Retrieval-Augmented Generation system?▼

Diagnosing RAG bottlenecks involves running end-to-end evaluations with golden answers to perform per-query analysis. This separates retrieval issues from generation issues, allowing you to identify exactly which stage causes poor performance and target improvements effectively.

Can I compare multiple RAG pipelines using golden answers and queries?▼

Yes, you can compare multiple RAG pipelines by applying cross-pipeline configuration comparisons. Supplying your queries and golden answers allows the evaluation to surface performance differences between configurations, highlighting variations in retrieval and generation metrics.

What metrics are used for RAG retrieval and generation evaluation?▼

RAG retrieval evaluation uses Recall, Precision, MRR, and NDCG to measure document fetching effectiveness. Generation evaluation uses Faithfulness, Relevance, and Completeness metrics to assess the final output quality against provided golden answers.

Do I need golden answers to run a RAG evaluation?▼

Yes, you need golden answers and evaluation queries to run a comprehensive RAG evaluation. These inputs are required to accurately measure generation fidelity and calculate retrieval metrics across your configured pipeline.