skills.rest
UKGovernmentBEIS
UKGovernmentBEISOFFICIAL
Explore all skills by this author →
GitHub

prepare-submission-workflow

Prepares an evaluation for register submission by generating eval.yaml and validating it.

657|419|Updated Oct 2, 2024
Search Tags:#evaluation#YAML#Validation#register#submission#Inspect Evals#Pull request
One-click install
npx skills add https://github.com/UKGovernmentBEIS/inspect_evals --skill prepare-submission-workflow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: prepare-submission-workflow
Source: https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/.claude/skills/prepare-submission-workflow
Command: npx skills add https://github.com/UKGovernmentBEIS/inspect_evals --skill prepare-submission-workflow

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Submitting a new evaluation to the Inspect Evals register requires verifying upstream repository requirements, authoring a correctly structured eval.yaml, running validation, and opening a compliant pull request, which is easy to get wrong manually.

Core Features & Use Cases

  • Upstream Repo Verification: Checks that the upstream repository has a pyproject.toml with a [project] table, declares inspect_ai as a dependency, and defines tasks with the @task decorator.
  • eval.yaml Generation: Derives fields like repository URL, commit SHA, task names, title, and description from the upstream repo, then drafts register/<eval_name>/eval.yaml using the example template.
  • Validation and PR Workflow: Runs make check to validate the YAML and auto-generate the README, creates a changelog fragment with scriv, and guides opening a PR with the repository template.
  • Use Case: A maintainer of an external evaluation repository wants to list it in the Inspect Evals register; this Skill walks them from repo verification through a ready-to-review pull request.

Quick Start

Ask the assistant to run the Prepare Evaluation For Submission workflow for your upstream evaluation repository URL.

Frequently Asked Questions about prepare-submission-workflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I submit a new evaluation to Inspect Evals?▼

New evaluations are submitted as register entries pointing to your own upstream repository, not as code in src/inspect_evals. You create register/<eval_name>/eval.yaml, run make check to validate it, and open a pull request.

What requirements must my upstream eval repository meet?▼

The upstream repo must have a pyproject.toml with a [project] table installable via uv sync, declare inspect_ai as a dependency, and define each task with the @task decorator from inspect_ai. You must also be the owner or a maintainer of that repository.

How do I create the eval.yaml file for the register?▼

Use register/example_eval.yaml as the template and derive fields from the upstream repo, such as repository URL, latest commit SHA, @task function names and paths, title, and description. Do not set the id field, since it is auto-injected from the directory name.

Can I edit the generated README in the register directory?▼

No, the README next to eval.yaml is fully generated by running make check and must not be edited by hand. Any documentation details should live in the upstream repository's README instead.

Why does my register submission validation fail?▼

Validation commonly fails when eval.yaml fields are missing or malformed, the commit is not a full 40-character SHA, or deprecated fields like contributors and tags are included. Run make check locally to see the specific errors before opening the PR.

Related Skills

Explore More Software Engineering Skills→
agenticsorgagenticsorg
25

github-multi-repo

Coordinate and synchronize multiple GitHub repositories using AI swarm orchestration.

Official
Advanced
GenerativeXGenerativeX

GH Review Comment Response

Automate pull request review comment resolution using the GitHub CLI.

Official
Intermediate
ice-droid-99ice-droid-99

agent-evaluation

Evaluate and improve LLM agent outputs using MLflow GenAI workflows.

Community
Advanced
UKGovernmentBEISUKGovernmentBEIS
617

code-quality-review-all

Audit evaluations under src/inspect_evals and generate results.json and SUMMARY.md.

Official
Advanced