skills.rest
MS33834
MS33834COMMUNITY
Explore all skills by this author →
GitHub

RAG Retrieval Eval

Creates comprehensive evaluation plans for document retrieval systems, including metrics, benchmarks, and analysis.

Updated Aug 27, 2026
Search Tags:#rag#benchmarking#metrics#evaluation#retrieval#reranking#datasets
One-click install
npx skills add https://github.com/MS33834/AI-SKILL --skill rag-retrieval-eval
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: RAG Retrieval Eval
Source: https://github.com/MS33834/AI-SKILL/tree/main/skills/rag-retrieval-eval
Command: npx skills add https://github.com/MS33834/AI-SKILL --skill rag-retrieval-eval

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Use this Skill when you need to determine whether a RAG retriever is finding the right evidence reliably instead of guessing from final answer quality alone.

Core Features & Use Cases

  • Metric selection: Define retrieval metrics such as recall, MRR, NDCG, precision, and latency targets.
  • Dataset design: Build golden query-to-document pairs, add hard negatives, and cover out-of-domain queries.
  • Baseline and iteration: Compare BM25 and embedding retrieval, then refine chunk size, overlap, embeddings, and reranking.
  • Use case: A team tuning support search or a knowledge base assistant can use this Skill to create a repeatable retrieval evaluation plan before changing the retriever.

Quick Start

Ask the Skill to design a retrieval evaluation plan for your corpus and representative queries, including metrics, dataset construction, baselines, and failure-mode checks.

Frequently Asked Questions about RAG Retrieval Eval

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG retrieval quality instead of relying on final answer quality?▼

To evaluate RAG retrieval quality, design a plan measuring recall, MRR, NDCG, and precision using golden query-document pairs, hard negatives, and latency targets to isolate retriever performance from generation.

How do I build a dataset for RAG retrieval evaluation?▼

Build a retrieval evaluation dataset by creating golden query-to-document pairs, adding hard negative samples, and including out-of-domain queries to thoroughly test retriever robustness.

Does this approach work for debugging technical document corpora retrieval?▼

Yes, retrieval evaluation applies to technical document corpora by establishing baseline comparisons, running failure-mode analysis, and performing human spot checks to debug and tune retriever reliability.

What is the best way to compare BM25 and embedding retrieval for a knowledge base assistant?▼

The best way to compare BM25 and embedding retrieval is establishing baseline metrics and iterating on chunking, embeddings, and reranking strategies guided by failure-mode analysis on representative queries.

When should I add reranking to my RAG retrieval pipeline?▼

Add reranking after establishing baseline retrieval metrics and analyzing failure modes, using iteration guidance to refine embeddings and chunking before applying reranking to improve precision.

Related Skills

Explore More Software Engineering Skills→
JhabbigJhabbig

embedding-strategies

Select embedding models and chunking strategies for semantic retrieval pipelines.

Community
Advanced
rukikotoorukikotoo
6

rag

Query the econometrics RAG knowledge base to retrieve relevant document chunks.

Community
Intermediate
cli-agent-speccli-agent-spec
5

cli-agent-evaluate

Evaluate a CLI tool against one CLI Agent Spec failure mode with a scored check.

Official
Intermediate
kinnerkarmanishkinnerkarmanish

eval-rag

Evaluate RAG pipelines by scoring retrieval and generation against gold references.

Community
Advanced