eval-quality-workflow

Fix or review a single Inspect AI evaluation against EVALUATION_CHECKLIST.md standards.

657|419|Updated Oct 2, 2024
One-click install
npx skills add https://github.com/UKGovernmentBEIS/inspect_evals --skill eval-quality-workflow
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: eval-quality-workflow
Source: https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/.claude/skills/eval-quality-workflow
Command: npx skills add https://github.com/UKGovernmentBEIS/inspect_evals --skill eval-quality-workflow

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Ensuring that community-contributed LLM evaluations in the inspect_evals repository comply with the project's quality standards is tedious and error-prone when done manually. This Skill automates the process of either refactoring an evaluation into compliance or reviewing it against the EVALUATION_CHECKLIST.md standards without making changes.

Core Features & Use Cases

  • Fix Mode: Walks through every item in EVALUATION_CHECKLIST.md, refactors the evaluation code to comply, tracks confidence levels, and records uncertainties in structured notes.
  • Review Mode: Assesses an evaluation against agent-runnable checklist items without modifying code, producing issue reports with locations, recommended fixes, and GitHub-ready comments.
  • Structured Artefacts: Organizes all output (NOTES.md, UNCERTAINTIES.md, SUMMARY.md) under agent_artefacts/<eval_name>/ for traceability.
  • Use Case: A maintainer asks to review the gpqa evaluation; the Skill checks each checklist item, documents issues with file and line references, and produces polite GitHub-compatible review comments.

Quick Start

Ask the assistant to run the Fix An Evaluation workflow on the gpqa evaluation in src/inspect_evals.

Frequently Asked Questions about eval-quality-workflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix an Inspect AI evaluation to meet quality standards?

Use the fix mode of this workflow, which walks through every item in EVALUATION_CHECKLIST.md top to bottom and refactors the evaluation code to comply. It records progress in NOTES.md, UNCERTAINTIES.md, and a final SUMMARY.md under agent_artefacts/<eval_name>/fix.

How do I review an evaluation without changing its code?

Use the review mode, which assesses the evaluation against the agent-runnable checks in EVALUATION_CHECKLIST.md without modifying files. It documents each issue with the standard violated, location, recommended fix, and a GitHub-ready comment prefixed with (Agent).

What is the difference between fix mode and review mode?

Fix mode actively refactors the evaluation to pass each checklist item and checks items off, while review mode only assesses compliance and reports issues. Both modes create the same artefact structure but under fix/ and review/ subfolders respectively.

Does this workflow generate the evaluation report automatically?

No, the evaluation report is explicitly out of scope. The workflow notes in SUMMARY.md that producing evaluation reports requires the separate /eval-report-workflow skill, though it still runs initial smoke tests to confirm the evaluation executes.

When should I not use the eval-quality-workflow skill?

Do not use it to review all evaluations against a single code quality standard; that scenario belongs to the code-quality-review-all skill. This workflow targets one evaluation against the full checklist.