Generate skill evals from mined conversation history and user corrections.
npx skills add https://github.com/garrytan/gbrain --skill skill-autobench
Please help me install this Agent Skill. Skill: skill-autobench Source: https://github.com/garrytan/gbrain/tree/main/skills/skill-autobench Command: npx skills add https://github.com/garrytan/gbrain --skill skill-autobench
Spec-derived benchmarks only test what a skill promised, not what users actually asked for or where it actually failed. This Skill authors evals from real usage history — mining conversation archives and session transcripts for invocations and corrections — so benchmarks reflect lived behavior instead of imagination.
Ask the agent to autobench a named skill, for example: autobench the media-ingest skill and write the eval from my usage history.
High-intent search queries and answers about installing and using this skill.
Run skill-autobench with the target skill name. It mines the brain's conversation archive and session transcripts for real invocations and corrections, then stages a proposed eval_contract with 4-8 replayable cases at skills/<name>/eval/autobench-<date>.md for human approval.
The skill fails closed: it emits an honest no-history report listing substrates checked and queries run, and never fabricates typical invocations. For skills with no history, use gbrain skillopt <name> --bootstrap-from-skill for a spec-derived benchmark instead.
No. It never rewrites SKILL.md, triggers, or any routing surface. The synthesized eval is staged with status PENDING-HUMAN-APPROVAL, and merging the approved eval_contract into frontmatter is an explicit human decision.
It asserts over the judging result object that each named model returned a non-empty response, that responses came from distinct provider endpoints, and that no two differently-named models returned byte-identical output. The check is pure assertion logic with no model calls.
All mined cases are privacy-scrubbed before staging: real people, companies, and deals are rewritten onto placeholder slugs like alice-example and acme-example. Cases keep their shape and failure mode but never their real entities.
skill-optimizer improves a skill's body against an existing benchmark, while skill-autobench authors the benchmark itself from lived usage and feeds it into skillopt-benchmark.jsonl. Use autobench when real invocation history exists; use skillopt --bootstrap-from-skill when it does not.
Nairon-AIMine archived conversations for uncaptured patterns and corrections.
jiangmin8Automate nightly skill optimization cycles with gated held-out scoring.
Impertio-StudioBuild responsive shadcn modals switching between Dialog and Drawer by viewport width.
evanfang0054Generate a risk-based test coverage tree with Level Items, Gates, and Assertions from a design spec.
