skills.rest
kwojtasinski-repo
kwojtasinski-repoCOMMUNITY
Explore all skills by this author →
GitHub

generate-eval-questions

Generates eval question JSON entries for newly indexed documentation files.

Updated Nov 19, 2020
Search Tags:#documentation#rag#json#question generation#qdrant#retrieval evaluation#eval questions
One-click install
npx skills add https://github.com/kwojtasinski-repo/ECommerceApp --skill generate-eval-questions-kwojtasinski-repo
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: generate-eval-questions
Source: https://github.com/kwojtasinski-repo/ECommerceApp/tree/main/.github/skills/generate-eval-questions
Command: npx skills add https://github.com/kwojtasinski-repo/ECommerceApp --skill generate-eval-questions-kwojtasinski-repo

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? When new documentation is indexed into the Qdrant vector store, the RAG evaluation suite in tools/rag/eval/questions.json has no coverage for it, leaving retrieval quality untested. This Skill produces a consistent, reviewable eval question entry for any newly indexed document. ## Core Features & Use Cases - Title Extraction: Reads the target document and derives its title from the first Markdown heading, falling back to the filename. - Path-Aware Question Styles: Applies per-directory question templates for ADRs, architecture docs, roadmap files, references, and agent context files. - Structured JSON Output: Emits an object with q, expect_any, auto_generated, and reviewed fields ready to append to questions.json. - Use Case: After indexing docs/adr/0017/0017-refund-policy-rules.md, generate the entry {"q": "How are the Refund Policy Rules designed?", "expect_any": ["docs/adr/0017/"]} so the /rag-sync agent can append it for eval scoring. ## Quick Start Generate an eval question entry for the newly indexed file docs/adr/0017/0017-refund-policy-rules.md.

Frequently Asked Questions about generate-eval-questions

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate an eval question for a newly indexed documentation file?▼

Provide the file path relative to the repo root. The Skill reads the document, extracts its title from the first Markdown heading, and returns a JSON object with a natural-language question and the expect_any path for the eval suite.

What question style is used for ADR documents?▼

Files under docs/adr/XXXX/ use the pattern "How is the [title] designed?", and expect_any points to the ADR folder path rather than the exact filename so it matches the main ADR and all amendments.

How is the document title extracted if there is no heading?▼

The Skill first looks for a level-one Markdown heading on any line. If none exists, it falls back to using the filename without its extension as the document title.

Where does the generated eval question get stored?▼

The /rag-sync agent appends the returned object to the questions array in tools/rag/eval/questions.json. The entry is shown to the user for confirmation before being used in eval scoring.

What are the limitations of the generated eval questions?▼

Generated entries are marked auto_generated true and reviewed false, meaning they require human confirmation before eval scoring. Question quality depends on the document having a meaningful heading or filename.

Related Skills

Explore More Software Engineering Skills→
softsparksoftspark
161

index

Index project source files and documentation into a vector store for semantic search.

Community
Intermediate
BryanPinheiro77BryanPinheiro77
2

senior-prompt-engineer

Analyze, optimize, and evaluate prompts, RAG pipelines, and agent configurations with stdlib Python tools.

Community
Advanced
EquiStampEquiStamp

evaluation-design

Design and review AI safety evaluations against the 7-dimension rubric.

Official
Advanced
TestAny-ioTestAny-io
79

brd-interviewer

Guide phase-based interviews to convert business ideas into a BRD draft.

Official
Advanced