skills.rest
breethomas
breethomasCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

eval-rag

Evaluates retrieval and generation quality for RAG pipelines using ground-truth document-chunk mappings.

16|3|Updated Oct 23, 2025
Search Tags:#recall#rag#evaluation#retrieval#generation#mrr#ndcg
One-click install
npx skills add https://github.com/breethomas/bette-think --skill eval-rag
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: eval-rag
Source: https://github.com/breethomas/bette-think/tree/main/plugins/pm-thought-partner/skills/eval-rag
Command: npx skills add https://github.com/breethomas/bette-think --skill eval-rag

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Diagnose and quantify where a retrieval-augmented generation (RAG) pipeline fails by separating retrieval quality from generation faithfulness so teams can prioritize the highest-impact fixes and avoid optimizing the wrong component.

Core Features & Use Cases

  • Retrieval metrics: Compute Recall@k, Precision@k, MRR, and NDCG@k to measure whether the system finds the right document chunks for a query.
  • Generation evaluation: Assess faithfulness, omissions, misinterpretations, and relevance of model outputs given retrieved context.
  • Optimization guidance: Build retrieval evaluation datasets, tune chunking and overlap, run grid searches, diagnose multi-hop failures, and produce prioritized engineering recommendations.
  • Use Case: A PM wants to know whether customer-support answers are failing because the vector store misses FAQ paragraphs or because the LLM hallucinates; this skill yields metrics, diagnostic tables, and next-step fixes.

Quick Start

Evaluate the RAG pipeline for the customer knowledge base and return retrieval metrics (Recall@k, Precision@k, MRR, NDCG@k), a faithfulness/relevance summary of generation failures, and recommended fixes.

Frequently Asked Questions about eval-rag

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate retrieval quality in a RAG pipeline?▼

Evaluate RAG retrieval quality by computing Recall@k, Precision@k, MRR, and NDCG@k against a dataset of queries with ground-truth document-chunk mappings. This separates search failure from generation issues to isolate where the pipeline drops relevant context.

Why does my RAG pipeline hallucinate or return irrelevant answers?▼

RAG pipelines fail when the vector store misses relevant chunks or the LLM hallucinates from retrieved context. Diagnose generation faithfulness and relevance separately from retrieval metrics to prioritize fixing the correct component.

What metrics are needed to diagnose multi-hop retrieval failures?▼

Diagnose multi-hop retrieval failures using Recall@k, Precision@k, MRR, and NDCG@k alongside multi-hop recall diagnostics. These metrics identify whether the system finds intermediate document chunks required to answer complex queries.

How do I tune chunk size and overlap for a vector store?▼

Tune chunking and overlap by running grid searches with a retrieval evaluation dataset. Measure Recall@k and NDCG@k across different chunk configurations to produce prioritized engineering recommendations for optimization.

Do I need ground-truth document mappings to evaluate generation faithfulness?▼

Evaluating generation faithfulness requires retrieved context and model outputs to assess omissions, misinterpretations, and relevance. Retrieval metrics like Recall@k additionally require ground-truth document-chunk mappings for your queries.

Related Skills

Explore More Product & Management Skills→
giuseppe-trisciuogliogiuseppe-trisciuoglio
322

rag-implementation

Build retrieval-augmented generation pipelines with vector databases and embedding models.

Community
Advanced
lagarcesslagarcess
1

rag-architect

Design and optimize retrieval-augmented generation pipelines for document indexing and vector search.

Community
Advanced
JayRHaJayRHa
3

rag-pipeline-designer

Design end-to-end RAG pipelines with chunking, embeddings, retrieval, and evaluation.

Community
Advanced
MantisWareMantisWare

eval-rag

Evaluate RAG pipelines with retrieval and generation metrics and bottleneck analysis.

Community
Advanced