skills.rest
SLEEPYBQ
SLEEPYBQCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

manage-evals

Trigger AI evaluation runs and compare results via GitHub workflows.

Updated Aug 27, 2026
Search Tags:#automation#performance#github#evaluation#comparison#model-testing
One-click install
npx skills add https://github.com/SLEEPYBQ/adaptive-rehearsal --skill manage-evals
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: manage-evals
Source: https://github.com/SLEEPYBQ/adaptive-rehearsal/tree/main/software-agent-sdk/.agents/skills/manage-evals
Command: npx skills add https://github.com/SLEEPYBQ/adaptive-rehearsal --skill manage-evals

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires urllib, json, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies the process of triggering, comparing, and reporting on AI evaluation runs across different benchmarks and models.

Core Features & Use Cases

  • Trigger Evaluations: Initiate new evaluation runs on various benchmarks with customizable configurations via GitHub workflows.
  • Compare Results: Fetch and analyze two evaluation runs to identify performance differences and regressions at the instance level.
  • Reporting and Commenting: Generate detailed markdown reports highlighting success rates, regressions, and improvements, with options to post comments directly on GitHub pull requests.

Quick Start

Trigger a new evaluation on a specific branch or tag, then compare the latest run with a previous baseline for performance insights.

Frequently Asked Questions about manage-evals

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate AI model evaluation comparison and trigger runs via GitHub?▼

You can automate AI model evaluation comparison by triggering new benchmark runs via GitHub workflows, then fetching and analyzing evaluation artifacts to identify performance differences and regressions at the instance level.

How do I report evaluation regressions and improvements directly on a GitHub pull request?▼

You can report evaluation regressions by generating detailed markdown reports highlighting success rates and improvements, with options to post these comparison comments directly on GitHub pull requests using the GitHub API.

What is needed to fetch and analyze evaluation results stored in cloud storage?▼

To fetch and analyze evaluation results, you need access to evaluation artifacts stored remotely in cloud storage and valid GitHub API credentials to trigger workflows and post comparison comments.

Does this evaluation management workflow support regression detection across different benchmarks?▼

Yes, regression detection is supported across different benchmarks by fetching two evaluation runs, analyzing their differences at the instance level, and documenting findings for review.

Can I use Python standard libraries to parse evaluation artifacts and generate markdown reports?▼

Yes, you can use Python standard libraries like urllib and json to fetch remote evaluation artifacts, parse the data, and generate detailed markdown reports for comparison and regression analysis.

Related Skills

Explore More Software Engineering Skills→
tanyagraytanyagray

bitbucket-pipeline

Manage Bitbucket Pipelines via the Bitbucket REST API.

Community
Advanced
ilamanovilamanov

fix-pr-reviews

Fetch and analyze GitHub PR review comments and markdown findings.

Community
Intermediate
kis9akis9a

issue-fix

Automate GitHub Issue resolution workflows for Go projects with PR generation.

Community
Advanced
WindChimeRanWindChimeRan
1

benchmark

Automate vLLM serving benchmarks and generate PR-ready markdown reports.

Community
Advanced