Trigger CI-based benchmark evaluations for the OpenHands software-agent-sdk suite.
npx skills add https://github.com/neubig/workflow --skill eval-with-ci
Please help me install this Agent Skill. Skill: eval-with-ci Source: https://github.com/neubig/workflow/tree/main/skills/eval-with-ci Command: npx skills add https://github.com/neubig/workflow --skill eval-with-ci
This skill enables teams to automate the execution of CI-based evaluations for the OpenHands/software-agent-sdk, streamlining benchmarking across multiple tools and repos.
Trigger a quick evaluation by labeling a PR with run-eval-50 or by dispatching the run-eval workflow.
High-intent search queries and answers about installing and using this skill.
You can trigger CI-based SWE-bench and GAIA evaluations on a pull request by applying a specific label like run-eval-50 or by manually dispatching the run-eval workflow within your GitHub Actions pipeline.
Yes, you can parameterize CI-based benchmark runs by configuring eval_limit to control the number of instances evaluated and specifying optional model_ids to target specific models during the workflow dispatch.
The best way to automate benchmark evaluations across SDK branches is to use configurable sdk_ref and branch controls within your CI workflow, dispatching the results automatically to the designated evaluation repository.
Yes, the CI-based evaluation orchestration supports triggering and monitoring Terminal-Bench alongside SWE-bench and GAIA benchmarks across pull requests and workflow runs.
Evaluation results are automatically dispatched and routed to the OpenHands evaluation repositories for review and aggregation immediately after the CI-based benchmark workflow completes.
You can parameterize the sdk_ref to point to a specific software-agent-sdk version or branch, but the workflow also supports default references when triggering evaluations via PR labels or manual dispatch.
fabnConfigure actionlint CI to lint GitHub Actions workflow files on push and pull request events.
DisentinelRuns A/B tests of the Grafema code graph on SWE-bench.
FutureAtomsMonitor and manage GitHub check statuses for pull requests and commits.
DiologIRRe-run whole-corpus quality reviews of diolog-swe-bench and diff against previous reviews.
