skills.rest
MikkoNumminen
MikkoNumminenCOMMUNITY
Explore all skills by this author →
GitHub

rag-audit

Validate RAG retrieval accuracy and refusal logic with Docker-based acceptance tests.

Updated Apr 6, 2026
Search Tags:#testing#validation#audit#llm#rag#retrieval
One-click install
npx skills add https://github.com/MikkoNumminen/mikkonumminen.dev --skill rag-audit-mikkonumminen
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-audit
Source: https://github.com/MikkoNumminen/mikkonumminen.dev/tree/main/.claude/skills/rag-audit
Command: npx skills add https://github.com/MikkoNumminen/mikkonumminen.dev --skill rag-audit-mikkonumminen

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill addresses the inconsistency and manual overhead of auditing RAG (Retrieval-Augmented Generation) systems by providing a standardized, adversarial battery of tests to ensure retrieval accuracy and gate integrity.

Core Features & Use Cases

  • Acceptance Testing: Validates the RAG system against a strict 9/9 acceptance contract to prevent regressions.
  • Adversarial Auditing: Includes specific test cases for containment leaks, generative task gates, and translation task gates to ensure the model refuses out-of-scope queries.
  • Performance Evaluation: Measures dense vs. hybrid retrieval hit-rates to optimize search quality.
  • Use Case: Use this before merging any RAG-related pull request to verify that changes to chunking, prompts, or retrieval logic do not introduce leaks or over-gating.

Quick Start

Run the rag-audit skill to execute the full acceptance battery and validate the current RAG backend against the local development environment.

Frequently Asked Questions about rag-audit

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate acceptance testing for my RAG system?▼

RAG validation uses automated acceptance testing and adversarial query batteries to verify retrieval accuracy, gate integrity, and refusal logic against a strict 9/9 contract to prevent regressions.

What is gate integrity in retrieval-augmented generation systems?▼

Gate integrity in retrieval-augmented generation ensures the model refuses out-of-scope queries, preventing containment leaks and blocking unauthorized generative or translation tasks through adversarial auditing.

Do I need Docker and Python to run RAG validation scripts?▼

Yes, RAG validation requires Docker and Python-based evaluation scripts to verify hit-rates and refusal logic against the local corpus within a containerized backend environment.

When should I run an adversarial audit on my RAG backend?▼

Run an adversarial RAG audit before merging any pull request to verify that changes to chunking, prompts, or retrieval logic do not introduce containment leaks or over-gating.

How do I measure dense vs hybrid retrieval hit-rates for search quality?▼

Measure dense vs hybrid retrieval hit-rates by executing Python evaluation scripts within a containerized backend, comparing search quality and accuracy directly against your local development corpus.

What are the limitations of automated RAG acceptance testing?▼

Automated RAG acceptance testing is limited to verifying hit-rates and refusal logic against a local corpus, requiring a containerized Docker backend and Python scripts to execute the validation battery.

Related Skills

Explore More Software Engineering Skills→
caseymanoscaseymanos

local-ssl

Automates creation and trust of local SSL certificates for development environments.

Community
Advanced
k1lgork1lgor
2

search-vector-architect

Implement hybrid search with Elasticsearch, Pinecone, Chroma, or pgvector.

Community
Advanced
JCETools-PetraJCETools-Petra
16

ai-llm-engineering

Design RAG pipelines with chunking, embedding, retrieval, and evaluation metrics.

Community
Advanced
BagelHoleBagelHole
46

rag-observability-evals

Monitor RAG systems with retrieval and generation quality metrics.

Community
Advanced