eval-recipes-runner

Automating benchmarking for pull request validation requires running the eval-recipes suite to quantify improvements across tasks. This Skill orchestrates CI validation and PR evaluation contexts to generate comparative reports against baseline runs.

The eval-recipes benchmark suite covers tasks including linkedin_drafting, email_drafting, arxiv_paper_summarizer, and github_docs_extractor. These tasks validate agent improvements across different contexts like research and PR review.

Yes, the uv workflow runner is required to orchestrate tasks when running eval-recipes benchmarks. You also need access to the eval-recipes repository and appropriate agent configurations for reproducible results.

Comparing benchmark scores against a baseline run involves executing eval-recipes for specific branches and tasks. The automation generates comparative reports to quantify changes made by amplihack improvements.

Yes, this automation applies CI validation contexts to evaluate agent configurations. It coordinates environment setup to ensure reproducible benchmarking results across different branches.

Quantifying improvements in amplihack is best achieved by automating benchmarks with the eval-recipes suite. This validates changes across multiple tasks and generates comparative reports against baseline scores.