evaluations

Name: evaluations
Availability: InStock
Author: vpittamp

Automates end-to-end evaluation workflows for datasets, evals, and runs.

2|Updated Aug 18, 2025

Search Tags:#provenance #evaluations #datasets #workflow-builder #graders #swe-bench #code-eval

Ready to run

npx skills add https://github.com/vpittamp/nixos-config --skill evaluations

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: evaluations
Source: https://github.com/vpittamp/nixos-config/tree/main/shared-skills/evaluations
Command: npx skills add https://github.com/vpittamp/nixos-config --skill evaluations

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Workflow-Builders need a unified platform to create, run, and grade evaluations across datasets, evals, and runs, with provenance for SWE-bench benchmarks and code-eval templates.

Core Features & Use Cases

End-to-end lifecycle for datasets, evals, runs, and run items with bidding grader configurations and provenance tracking.
Supports code-eval templates (HumanEval+/MBPP+/BigCodeBench) and SWE-bench style benchmarks, with per-agent grading and async/sync graders.
Public UI/API surfaces for managing datasets, evals, runs, and benchmark provenance across templates and workflows.

Quick Start

Create a new evaluation run from the UI or API to seed a dataset and begin grading.

Frequently Asked Questions about evaluations

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run end-to-end evaluations across datasets and code-eval templates?▼

End-to-end evaluations are automated by building, running, and grading workflow-builder runs across datasets and evals. You create a new run from the UI or API to seed a dataset and begin grading with configured graders.

Can I grade SWE-bench benchmarks with async graders and provenance tracking?▼

SWE-bench benchmarks support async graders, per-agent runtimes, and dedicated provenance tables. The system stores benchmark provenance across code-eval templates and workflow-driven runs for accurate benchmark tracking.

What code-eval templates are supported for running dataset evaluations?▼

Supported code-eval templates include HumanEval+, MBPP+, and BigCodeBench. These templates integrate into the workflow-builder lifecycle to manage datasets, evals, and run items with both sync and async graders.

Does the evaluation workflow require a specific data structure to operate?▼

A structured model containing datasets, evals, and run items is required. This structure supports UI-driven wizard flows, API surfaces, and per-agent grading configurations to coordinate the full evaluation lifecycle.

What's the best way to track provenance across workflow-builder evaluations and runs?▼

Provenance is tracked by storing dataset, eval, and run history in dedicated provenance tables. This applies to both SWE-bench style benchmarks and code-eval templates, ensuring full lineage across the evaluation lifecycle.

Why do my code-eval runs need per-agent runtimes for grading?▼

Per-agent runtimes isolate grading logic for each evaluation run item, allowing both sync and async graders to execute reliably. This ensures consistent benchmark results across SWE-bench and code-eval template workflows.