aeon-skill-evals

To validate skill output against an assertion manifest, you parse YAML manifests and compare output data against criteria like word counts, patterns, sections, and citations. This quality assurance process confirms outputs meet specifications before deployment.

To detect regressions in skill development, you compare current skill outputs against past successful runs. By identifying deviations from previous baseline behaviors, you can pinpoint regressions and maintain quality assurance across iterative skill updates.

A YAML assertion manifest is a configuration file defining expected output criteria like word counts, patterns, and citations. You need it for skill validation when establishing quality gates to systematically verify output data and detect regressions during development.

You create assertion manifests for new skills by bootstrapping them from recent successful runs. This avoids speculative writing by extracting actual output patterns, word counts, and sections from proven valid data to generate reliable validation criteria.

Yes, you can check skill output against quality gates automatically using assertion manifests. The system parses YAML manifests to compare output data against predefined criteria, triggering validation checks that detect regressions and ensure quality control.

Limitations when validating output against manifest criteria include dependency on accurate YAML manifest creation and the ability to parse output data correctly. Validation only checks defined criteria like patterns and citations, potentially missing semantic quality issues.