What problem does it solve?
Writing and improving AI agent skills is usually guesswork: you edit a prompt, eyeball a few outputs, and ship. This Skill turns skill authoring into a measurable engineering loop—author, evaluate across models and harnesses, optimize with pre-registered hypotheses, and release with regression evidence—so every change to a skill is backed by tagged, reproducible measurements.
Core Features & Use Cases
- Four-stage lifecycle: author, eval, optimize, and release stages, each with its own protocol reference and a state machine that resumes from repository facts in
experiments/.
- Deterministic evaluation harness: bundled runner, scorers (planted-defect recall, DoD gate, LLM judge), and prompt builders execute a skill snapshot against fixtures across multiple models and harnesses with cost guardrails.
- Cognitive honesty enforcement: every number carries a
[measured], [soft], or [underpowered] tag; hypotheses must be git-committed before any LLM run; judge models must be independent from tested models.
- Use Case: You rewrote the
cs-review skill and want to know if it actually catches more defects. Run the eval stage against planted-defect fixtures on two model families, then let the optimize loop test variants until the convergence criteria are met, and release with a version bump and regression battery.
Quick Start
Ask the agent to evaluate the cs-review skill with eval-cs-skill, starting with a dry-run cost estimate before running the full experiment.
