skills.rest
Mathews-Tom
Mathews-TomCOMMUNITY
Explore all skills by this author →
𝕏@tom_mathewsGitHub

rag-auditor

Evaluate RAG pipeline retrieval and generation quality with metrics.

310|45|Updated Feb 22, 2026
Search Tags:#embeddings#rag#evaluation#retrieval#grounding#hallucination#mrr
One-click install
npx skills add https://github.com/Mathews-Tom/praxis-skills --skill rag-auditor
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-auditor
Source: https://github.com/Mathews-Tom/praxis-skills/tree/main/skills/rag-auditor
Command: npx skills add https://github.com/Mathews-Tom/praxis-skills --skill rag-auditor

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

RAG Auditor provides a repeatable, measurable way to find why retrieval-augmented generation systems return incorrect, incomplete, or hallucinated answers by separating retrieval, chunking, embedding, ranking, and generation issues and quantifying their impact.

Core Features & Use Cases

  • Retrieval evaluation: Measures Precision@K, Recall@K, MRR, Hit@K and produces per-query relevance judgments to pinpoint retrieval and ranking failures.
  • Generation assessment: Quantifies groundedness, completeness, hallucination rate, and abstention accuracy to determine whether errors come from the generator versus the retriever.
  • Failure diagnosis & recommendations: Classifies failures (retrieval, chunking, embedding, generation) and recommends targeted fixes such as chunking strategies, embedding swaps, reranking, or prompt grounding rules.
  • Use Case: Use when a production RAG system returns confident but unsupported answers or when you need a prioritized remediation plan to improve knowledge-grounded responses.

Quick Start

Run a RAG audit by providing a representative set of known-answer queries, the retrieved chunk lists and scores for each query, the generator outputs, and basic pipeline metadata.

Frequently Asked Questions about rag-auditor

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate RAG pipeline retrieval quality and measure Precision@K?▼

Evaluate RAG pipeline retrieval quality by providing test queries with known answers alongside retrieved chunk lists and scores to compute Precision@K, Recall@K, MRR, and Hit@K for per-query relevance judgments.

Why does my retrieval-augmented generation system return hallucinated answers?▼

Hallucinated answers occur when RAG generation lacks grounding. Audit the generator by quantifying groundedness, completeness, hallucination rate, and abstention accuracy to determine if errors stem from the generator or retriever.

How do I diagnose RAG chunking and embedding failures in a production system?▼

Diagnose RAG chunking and embedding failures by running an audit that classifies errors across retrieval and generation stages, then apply targeted fixes like chunking strategy adjustments, embedding swaps, or reranking.

What data do I need to run a RAG audit for grounding and hallucination rates?▼

Running a RAG audit requires a representative set of known-answer queries, retrieved chunk lists with scores, generator outputs, and optional corpus metadata to compute grounding and hallucination rates.

Can I use RAG auditing to compare different embedding models and ranking strategies?▼

Yes, RAG auditing evaluates retrievers, chunking strategies, embedding models, and ranking by measuring MRR and Hit@K across test queries to identify retrieval and ranking failures for targeted remediation.

What is the best way to find why a RAG system returns incomplete or unsupported answers?▼

The best way to find why a RAG system returns unsupported answers is auditing retrieval and generation stages separately to quantify grounding, completeness, and retrieval precision for a prioritized remediation plan.

Related Skills

Explore More Data & Analytics Skills→
floflo777floflo777
33

rag-eval

Evaluate RAG systems using retrieval, generation, and latency metrics.

Community
Intermediate
Sir-chawakornSir-chawakorn
9

rag-pipeline

Build and tune retrieval-augmented-generation pipelines for private corpora.

Community
Advanced
ricardorochericardoroche
2

rag-design-patterns

Standardizes chunking, vector database integration, embedding management, and retrieval reranking for RAG pipelines.

Community
Intermediate
ivanshamaevivanshamaev
14

rag-data-pipeline

Build RAG ingestion and query pipelines with hybrid retrieval and reranking.

Community
Advanced