building-rag-eval-set

What problem does it solve?

This Skill turns a raw document corpus into a defensible evaluation set for retrieval-augmented generation, so teams can stop relying on ad-hoc questions and guesswork.

Core Features & Use Cases

• Greenfield evaluation design: Creates calibration, held-out, and adversarial splits instead of a single flat test set.
• Failure attribution: Requires source document IDs and source spans on every non-absent-topic question so retrieval errors can be separated from generation errors.
• Human-reviewed gold data: Forces review and ground-truth verification before any row enters the golden set, reducing noise and contamination.
• Production discipline: Locks the set with versioning, dataset hashing, and a documented review protocol for repeatable downstream audits.
• Use case: Build a proper RAG eval for internal HR policies, legal documents, medical corpora, or other custom domains where public benchmarks do not fit.

Quick Start

Ask the Skill to build a RAG evaluation set for your corpus and specify your reviewers, target size, and whether you need calibration, held-out, and adversarial splits.