skills.rest
davepoon
davepoonCOMMUNITY
Explore all skills by this author →
Website𝕏@davepoonGitHub

rag-evaluation-harness

Evaluate RAG retrieval and citation metrics from deterministic JSONL fixtures offline.

3.4k|487|Updated Jul 25, 2025
Search Tags:#evaluation#JSONL#rAG#CI testing#citation validity#recall@k#retrieval metrics
One-click install
npx skills add https://github.com/davepoon/buildwithclaude --skill rag-evaluation-harness
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-evaluation-harness
Source: https://github.com/davepoon/buildwithclaude/tree/main/plugins/all-skills/skills/rag-evaluation-harness
Command: npx skills add https://github.com/davepoon/buildwithclaude --skill rag-evaluation-harness

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve?

Measuring retrieval quality and citation behavior in RAG pipelines usually requires model calls or network access, making results non-repeatable and unsuitable for CI. This Skill computes Recall@K, reciprocal rank, context precision, citation coverage, and citation validity from explicit document IDs in JSONL fixtures, fully offline and deterministically.

Core Features & Use Cases

  • Offline Metric Computation: Calculates Recall@K, reciprocal rank, context precision@K, citation coverage, and citation validity by comparing explicit document ID arrays, with no model calls or network requests.
  • CI Threshold Gating: Accepts minimum thresholds for each metric and exits with code 1 when any threshold fails, enabling quality gates in continuous integration pipelines.
  • Diagnostics and Reporting: Flags empty retrieval, missing relevance labels, duplicate retrieved IDs, unretrieved citations, and missing citations, then emits Markdown or JSON reports.
  • Use Case: A team maintaining a RAG search feature commits a JSONL fixture of questions with known relevant documents, then runs the evaluator in CI with --min-recall 0.8 to block regressions in retrieval quality.

Quick Start

Run the bundled evaluator script with Node against a JSONL fixture file, specifying a K value and optional metric thresholds, to get a Markdown or JSON evaluation report.

Frequently Asked Questions about rag-evaluation-harness

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG retrieval quality offline?▼

Create a JSONL file where each line contains an id plus relevant_document_ids, retrieved_document_ids, and cited_document_ids arrays, then run the evaluate-rag.mjs script with Node. It computes Recall@K, reciprocal rank, and context precision without any model calls or network access.

How to gate RAG evaluation metrics in CI pipelines?▼

Pass threshold flags such as --min-recall, --min-mrr, or --min-citation-validity with values between 0 and 1. The process exits with code 1 when any threshold fails and writes failures to stderr, while the full report remains on stdout in JSON or Markdown format.

What metrics does a RAG evaluation harness compute?▼

It computes Recall@K, reciprocal rank, context precision@K, citation coverage, and citation validity, all macro-averaged across cases. Cases with no relevant documents contribute null to recall and coverage and are excluded from those averages.

Does RAG evaluation require API calls or external dependencies?▼

No, the evaluator uses only the Node.js standard library and reads a local JSONL file. It never executes retrieved content, calls MCP servers, accesses credentials, or mutates the input, making it fully deterministic and offline.

What are the limitations of ID-based RAG evaluation metrics?▼

ID-level metrics are proxies that do not measure semantic answer quality, entailment, attribution correctness, or groundedness. Pair them with a separate answer-quality evaluation when those properties matter for your use case.

Related Skills

Explore More Software Engineering Skills→
lofibrainwavlofibrainwav

ragas-evaluator

Score RAG output fidelity and relevance using four metrics.

Community
Intermediate
maziyarpanahimaziyarpanahi
5.0k

gating-deid-leakage

Fail CI when PHI recall drops below approved thresholds or critical identifiers leak.

Community
Advanced
NOMARJNOMARJ

llm-evaluation

Evaluate LLM applications with automated metrics, human feedback, and benchmarking.

Official
Advanced
Mercurium-AnalyticsMercurium-Analytics

hybrid-lexical-semantic

Fuse BM25 lexical relevance with vector semantic similarity in pgsv.hybrid_search.

Official
Advanced