Automates end-to-end evaluation workflows for datasets, evals, and runs.
npx skills add https://github.com/vpittamp/nixos-config --skill evaluations
Please help me install this Agent Skill. Skill: evaluations Source: https://github.com/vpittamp/nixos-config/tree/main/shared-skills/evaluations Command: npx skills add https://github.com/vpittamp/nixos-config --skill evaluations
Workflow-Builders need a unified platform to create, run, and grade evaluations across datasets, evals, and runs, with provenance for SWE-bench benchmarks and code-eval templates.
Create a new evaluation run from the UI or API to seed a dataset and begin grading.
High-intent search queries and answers about installing and using this skill.
End-to-end evaluations are automated by building, running, and grading workflow-builder runs across datasets and evals. You create a new run from the UI or API to seed a dataset and begin grading with configured graders.
SWE-bench benchmarks support async graders, per-agent runtimes, and dedicated provenance tables. The system stores benchmark provenance across code-eval templates and workflow-driven runs for accurate benchmark tracking.
Supported code-eval templates include HumanEval+, MBPP+, and BigCodeBench. These templates integrate into the workflow-builder lifecycle to manage datasets, evals, and run items with both sync and async graders.
A structured model containing datasets, evals, and run items is required. This structure supports UI-driven wizard flows, API surfaces, and per-agent grading configurations to coordinate the full evaluation lifecycle.
Provenance is tracked by storing dataset, eval, and run history in dedicated provenance tables. This applies to both SWE-bench style benchmarks and code-eval templates, ensuring full lineage across the evaluation lifecycle.
Per-agent runtimes isolate grading logic for each evaluation run item, allowing both sync and async graders to execute reliably. This ensures consistent benchmark results across SWE-bench and code-eval template workflows.
vm0-aiRun Apify Actors and retrieve dataset items via API.
DiologIRRe-run whole-corpus quality reviews of diolog-swe-bench and diff against previous reviews.
neubigTrigger CI-based benchmark evaluations for the OpenHands software-agent-sdk suite.
dyCuong03Draft, test, grade, and optimize Claude Code skills with benchmark evaluations.
