skill-eval-improve

Improves Agent Skills through layered evaluation, bounded edits, and held-out gates.

2|Updated Aug 15, 2026
One-click install
npx skills add https://github.com/Arenukvern/shortly --skill skill-eval-improve-arenukvern
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: skill-eval-improve
Source: https://github.com/Arenukvern/shortly/tree/main/.agents/skills/skill-eval-improve
Command: npx skills add https://github.com/Arenukvern/shortly --skill skill-eval-improve-arenukvern

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Agent Skills can drift over time: descriptions route incorrectly, SKILL.md files bloat, and edits overfit to a single failure. This Skill provides a measurable loop—baseline, evaluate, bounded edit, re-validate—so skill quality improves without destroying working rules. ## Core Features & Use Cases - Layered evaluation stack: Runs validator gates, rule-based YAML eval cases, Codex plugin-eval static analysis, human prompt suites, and SkillOpt-style bounded edit loops, choosing the cheapest layer that answers the question. - T1/T2 quality gates: Applies T1 behavior-critical CI checks (pnpm run eval with evals/cases/*.yaml) versus T2 structural validate-only checks per ADR 0011 and ADR 0027. - Held-out gating: Splits prompt suites into train and held-out sets, keeping an edit only if held-out results improve, and logs outcomes in references/evals.md. - Use Case: After reviewers report a skill's description is vague and routing fails, run layer 0 validation, add rule-based eval cases, apply a bounded edit under 10% line churn, and re-run held-out prompts before merging. ## Quick Start Ask the agent to improve a specific skill after routing eval failures and suggest one bounded quality fix with a held-out gate.

Frequently Asked Questions about skill-eval-improve

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I improve an Agent Skill after routing failures?▼

Start with layer 0 validation using pnpm run validate, then run rule-based eval cases with pnpm run eval. Apply a bounded edit of at most 10% line churn to the name or description, and keep the change only if held-out prompts improve.

What is the difference between T1 and T2 skill quality gates?▼

T1 behavior-critical skills require rule-based YAML eval cases plus pnpm run eval in CI because drift can change agent decisions. T2 structural skills only need pnpm run validate. The tiers are defined in ADR 0011 and ADR 0027.

Can I use Codex plugin-eval with Skill Steward skills?▼

Yes, when plugin-eval is installed locally you can run plugin-eval analyze, explain-budget, and init-benchmark against a skill directory. It complements but does not replace the validator and rule-based eval layers.

When should I not use the skill-eval-improve workflow?▼

Do not use it for bulk repo validation, which only needs pnpm run validate, or for creating a new skill, which belongs to skill-authoring-lifecycle. It also does not run automated SkillOpt cluster training; the bounded-edit loop is manual.

Why should LLM judges stay out of CI for skill evals?▼

LLM judges introduce flakiness and cost in CI, so ADR 0011 keeps them offline. CI runs only deterministic rule-based YAML cases, while subjective rubric scoring lives in references/evals.md as a human or offline review step.