skills.rest
kwojtasinski-repo
kwojtasinski-repoCOMMUNITY
Explore all skills by this author →
GitHub

rag-eval-coverage

Detect corpus files lacking eval query coverage and propose new queries.

Updated Nov 19, 2020
Search Tags:#documentation#rag#retrieval#gap analysis#eval coverage#parity audit#queries.yaml
One-click install
npx skills add https://github.com/kwojtasinski-repo/ECommerceApp --skill rag-eval-coverage-kwojtasinski-repo
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: rag-eval-coverage
Source: https://github.com/kwojtasinski-repo/ECommerceApp/tree/main/.github/skills/rag-eval-coverage
Command: npx skills add https://github.com/kwojtasinski-repo/ECommerceApp --skill rag-eval-coverage-kwojtasinski-repo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml.

What problem does it solve? Retrieval quality audits only measure files that named queries in queries.yaml actually target, leaving large parts of the documentation corpus with unmeasured retrieval quality. This Skill walks the corpus, compares every file against the expected results of all queries, and produces a gap report so you know exactly which files have zero eval coverage. ## Core Features & Use Cases - Coverage Gap Detection: Enumerates all Markdown files in the corpus, extracts covered paths from queries.yaml, and computes the uncovered set with shell or PowerShell commands. - Prioritised Query Proposals: Classifies uncovered files by path pattern (ADRs, architecture docs, roadmap, playbooks) and drafts new queries in the queries.yaml schema for high and medium priority files. - Audit Integration: Reminds you to register new query IDs in compare_queries.py slices so parity audits actually pick them up. - Use Case: After merging ADR-0030 with three new files, run this Skill to confirm which files lack coverage, draft two targeted queries, append them to queries.yaml, and verify coverage count rises accordingly. ## Quick Start Run the rag-eval-coverage skill against docs and .github/context to list uncovered files and draft queries for the high-priority ones.

Frequently Asked Questions about rag-eval-coverage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find files with no eval query coverage in a RAG corpus?▼

Enumerate all Markdown files in the corpus, extract every path listed in the expected results of queries.yaml, then compute the set difference with comm or PowerShell. Files in the corpus but absent from the covered list have zero coverage.

How do I write good eval queries for retrieval testing?▼

Draft a natural-language question whose answer lives in the target file, using a distinguishing keyword from that file. Avoid queries so specific they match one chunk or so vague they match everything, and record the target path and section hint in queries.yaml.

Does this skill work on Windows without Unix tools like find and comm?▼

Yes, the workflow includes PowerShell equivalents using Get-ChildItem for corpus enumeration and Where-Object for set difference. The Python coverage extraction script is cross-platform.

Why does a covered file still fail retrieval in the parity audit?▼

Coverage only checks that a query targets the file, not that the file was ingested or ranks well. Run the ingest script for new files, and use a ranking-tuning or query-debugging skill when a covered file returns the wrong chunk.

When should I not run a coverage gap analysis?▼

Skip it when coverage was checked recently and no docs were added, or when queries.yaml is empty or absent. Also skip it for a single file you already know is uncovered and write the query directly.

Related Skills

Explore More Software Engineering Skills→
Nice-Wolf-StudioNice-Wolf-Studio

wolf-adr

Index Architecture Decision Records across development phases for topic-based search.

Official
Basic
jonaseck2jonaseck2
2

gazetteer-testing

Audit genealogy place resolution against a SQLite database and bucket outlier mismatches.

Community
Advanced
hdkz-devhdkz-dev

ask-questions-if-underspecified

Ask clarifying questions when a request is underspecified before acting.

Community
Intermediate
JamesFincherJamesFincher

peft-fine-tuning

Train low-rank adapters for parameter-efficient fine-tuning of large language models.

Community
Advanced