Compare modified AI skills against git HEAD baselines with automated grading.
npx skills add https://github.com/vltansky/skills --skill skill-ab-eval
Please help me install this Agent Skill. Skill: skill-ab-eval Source: https://github.com/vltansky/skills/tree/main/skills/skill-ab-eval Command: npx skills add https://github.com/vltansky/skills --skill skill-ab-eval
This Skill helps you confidently make changes to your AI agent's skills by providing a rigorous way to measure whether those changes actually improve performance, preventing regressions and ensuring progress.
roast-my-agents-md skill, use this Skill to run it against a set of predefined prompts and assertions. The report will tell you if your changes made it better, worse, or had no effect compared to the original version.Use the skill-ab-eval skill to test your recent changes to the roast-my-agents-md skill.
High-intent search queries and answers about installing and using this skill.
You can benchmark AI agent skill changes by running A/B tests that compare modified versions against git HEAD baselines. The tool orchestrates parallel agent runs and automated output grading to generate interactive HTML reports with pass rates and overall verdicts.
Automated regression testing for skill development works by comparing new skill versions against baseline versions using predefined prompts and assertions. It orchestrates parallel agent runs and grades outputs automatically to detect regressions or improvements before deployment.
Yes, you can collect user feedback during AI agent A/B testing. The generated interactive HTML benchmark reports include user feedback collection features alongside pass rates and deltas to provide quantifiable validation of skill modifications.
The best way to validate skill improvements is to run rigorous A/B tests against predefined assertions. This process compares modified skills to git HEAD baselines and generates an overall verdict of improvement, regression, no change, or mixed results.
Yes, you need git to run these A/B tests because the evaluation compares modified skill versions against their git HEAD baselines. This dependency allows the tool to establish a controlled baseline for accurate regression testing.
A limitation of evaluating skill modifications with automated assertions is that grading relies strictly on predefined test cases. Complex or subjective skill behaviors not covered by the assertions may not be accurately captured in the HTML benchmark reports.
ckyeonCompare pinned commits against upstream HEAD and investigate tracked path changes.
PaddlePaddleBenchmark FastDeploy versus SGLang with end-to-end tests and HTML reports.
open-nekoCreate and iteratively improve AI agent skills through structured evaluation and benchmarking.
deveshkumarsMonitor production URLs post-deploy with canary checks and baseline comparisons.