keyhac-skill-cleanroom

Runs clean-room tests of packaged Keyhac skill bundles and records unanswered questions.

6|Updated Jul 26, 2026
One-click install
npx skills add https://github.com/crftwr/keyhac --skill keyhac-skill-cleanroom-crftwr
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: keyhac-skill-cleanroom
Source: https://github.com/crftwr/keyhac/tree/main/.claude/skills/keyhac-skill-cleanroom
Command: npx skills add https://github.com/crftwr/keyhac --skill keyhac-skill-cleanroom-crftwr

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) components.

What problem does it solve? It verifies whether a packaged Keyhac skill bundle is truly self-contained by having an isolated AI session write a real Keyhac action using only the bundle, recording every question the bundle could not answer as evidence of documentation gaps. ## Core Features & Use Cases - Isolated room construction: Builds a clean-room directory containing only the skill bundles, RULES.md, TASK.md, and QUESTIONS.md, with a canary CLAUDE.md that detects contaminated sessions. - Non-interactive session driving: Launches a restricted Claude session with scoped permissions, a single Keyhac MCP endpoint, and no Bash, so the room cannot reach the repository or ask the operator questions. - Transcript auditing and scoring: Audits the session transcript for rule breaches (reading the checkout, installed package, or other authors' actions), runs mechanical checks, and reports QUESTIONS.md as the actual result. - Use Case: After editing the keyhac-action-authoring skill, run make cleanroom CASE=1 to confirm the rebuilt bundle teaches a fresh session everything it needs, then fix the skill for every question recorded. ## Quick Start Ask the assistant to clean-room test the skill bundle with case 1 by running make cleanroom CASE=1 after enabling Keyhac's MCP Server switch.

Frequently Asked Questions about keyhac-skill-cleanroom

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a clean-room test of a Keyhac skill bundle?

Run `make cleanroom CASE=1` for a predefined eval case or `make cleanroom TASK="..."` for a custom task. The command rebuilds the bundles, creates an isolated room under cleanroom/, drives a non-interactive session, audits the transcript, and prints QUESTIONS.md.

What does a clean-room test of a skill bundle measure?

It measures whether the skill bundle alone can teach a fresh session to write a working Keyhac action. Every question the session had to guess at is recorded in QUESTIONS.md, and each entry identifies a gap the skill documentation should fill.

Why does the clean-room run fail with an MCP endpoint error?

Keyhac's MCP server switch (AI Integration > MCP Server) must be enabled before running. It is off by default and turns itself off 60 minutes after being ticked, so the run checks the endpoint first and refuses by name if it is off.

Can I drive a clean-room session interactively instead of automatically?

Yes, run `make cleanroom CASE=1 ARGS=--dry-run` to print the exact session command, paste it, and say only "Read RULES.md, then do TASK.md." You must answer nothing the session asks, and afterwards run score_room.py on the room and remove the test action from ~/.keyhac/extensions/.

What disqualifies a clean-room run?

A run is disqualified if the transcript shows the session reading the Keyhac checkout, the installed keyhac package source, or another author's action in ~/.keyhac/extensions/. Isolation failures like Bash being present or the checkout's skills leaking also invalidate the measurement.

Is zero questions in QUESTIONS.md a passing result?

No. A run with nothing recorded either measured nothing or broke a rule, so the transcript must be reviewed before trusting it. The judgement half is comparing the case's "must" list in evals/cases.md against the produced action.