rag-auditor

What problem does it solve?

RAG Auditor provides a repeatable, measurable way to find why retrieval-augmented generation systems return incorrect, incomplete, or hallucinated answers by separating retrieval, chunking, embedding, ranking, and generation issues and quantifying their impact.

Core Features & Use Cases

• Retrieval evaluation: Measures Precision@K, Recall@K, MRR, Hit@K and produces per-query relevance judgments to pinpoint retrieval and ranking failures.
• Generation assessment: Quantifies groundedness, completeness, hallucination rate, and abstention accuracy to determine whether errors come from the generator versus the retriever.
• Failure diagnosis & recommendations: Classifies failures (retrieval, chunking, embedding, generation) and recommends targeted fixes such as chunking strategies, embedding swaps, reranking, or prompt grounding rules.
• Use Case: Use when a production RAG system returns confident but unsupported answers or when you need a prioritized remediation plan to improve knowledge-grounded responses.

Quick Start

Run a RAG audit by providing a representative set of known-answer queries, the retrieved chunk lists and scores for each query, the generator outputs, and basic pipeline metadata.