skill-eval

A tiered evaluation framework measures AI agent skill quality across three levels: Tier 1 structural lint and trigger coverage, Tier 2 LLM-based semantic judging, and Tier 3 A/B effectiveness testing to drive iterative improvements.

Run a Tier 1 structural lint to score content depth and validate trigger coverage, then use LLM semantic evaluation to judge trigger alignment, followed by an A/B effectiveness workflow to iteratively improve trigger coverage.

LLM evaluation for semantic trigger alignment uses a large language model as a Tier 2 judge to assess how well skill triggers and descriptions match their intended purpose, ensuring reliable semantic quality beyond basic structural checks.

The end-to-end improvement workflow follows a diagnose, research, fix, and validate cycle, using GEPA-driven A/B effectiveness testing at Tier 3 to iteratively enhance skill quality and ensure consistent reliability across a fleet.

Yes, you can audit a fleet of AI agent skills for consistent reliability by applying a 3-tier evaluation framework that validates baseline structure, semantically judges trigger alignment, and drives GEPA-driven improvement workflows across the entire skill set.