rag-audit

Name: rag-audit
Availability: InStock
Author: mikkonumminen

Validate RAG retrieval accuracy and refusal logic with Docker-based acceptance tests.

Updated Apr 6, 2026

Search Tags:#testing #validation #audit #llm #rag #retrieval

Ready to run

npx skills add https://github.com/MikkoNumminen/mikkonumminen.dev --skill rag-audit-mikkonumminen

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: rag-audit
Source: https://github.com/MikkoNumminen/mikkonumminen.dev/tree/main/.claude/skills/rag-audit
Command: npx skills add https://github.com/MikkoNumminen/mikkonumminen.dev --skill rag-audit-mikkonumminen

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill addresses the inconsistency and manual overhead of auditing RAG (Retrieval-Augmented Generation) systems by providing a standardized, adversarial battery of tests to ensure retrieval accuracy and gate integrity.

Core Features & Use Cases

Acceptance Testing: Validates the RAG system against a strict 9/9 acceptance contract to prevent regressions.
Adversarial Auditing: Includes specific test cases for containment leaks, generative task gates, and translation task gates to ensure the model refuses out-of-scope queries.
Performance Evaluation: Measures dense vs. hybrid retrieval hit-rates to optimize search quality.
Use Case: Use this before merging any RAG-related pull request to verify that changes to chunking, prompts, or retrieval logic do not introduce leaks or over-gating.

Quick Start

Run the rag-audit skill to execute the full acceptance battery and validate the current RAG backend against the local development environment.

Frequently Asked Questions about rag-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate acceptance testing for my RAG system?▼

RAG validation uses automated acceptance testing and adversarial query batteries to verify retrieval accuracy, gate integrity, and refusal logic against a strict 9/9 contract to prevent regressions.

What is gate integrity in retrieval-augmented generation systems?▼

Gate integrity in retrieval-augmented generation ensures the model refuses out-of-scope queries, preventing containment leaks and blocking unauthorized generative or translation tasks through adversarial auditing.

Do I need Docker and Python to run RAG validation scripts?▼

Yes, RAG validation requires Docker and Python-based evaluation scripts to verify hit-rates and refusal logic against the local corpus within a containerized backend environment.

When should I run an adversarial audit on my RAG backend?▼

Run an adversarial RAG audit before merging any pull request to verify that changes to chunking, prompts, or retrieval logic do not introduce containment leaks or over-gating.

How do I measure dense vs hybrid retrieval hit-rates for search quality?▼

Measure dense vs hybrid retrieval hit-rates by executing Python evaluation scripts within a containerized backend, comparing search quality and accuracy directly against your local development corpus.

What are the limitations of automated RAG acceptance testing?▼

Automated RAG acceptance testing is limited to verifying hit-rates and refusal logic against a local corpus, requiring a containerized Docker backend and Python scripts to execute the validation battery.