What problem does it solve?
Creating a credible evaluation report for an Inspect AI eval requires selecting suitable models, estimating compute costs, running evaluations within a budget, and formatting results consistently. This Skill automates that entire workflow so contributors can produce standardized, reproducible reports for the inspect_evals repository.
Core Features & Use Cases
- Model Selection & Cost Estimation: Picks paper-relevant or frontier models from multiple providers and estimates token usage and dollar cost from a small pilot run, capped at $5 per model.
- Automated Report Generation: Drives tools/evaluation_report.py to produce a report.md with results tables, reference comparisons, per-category breakdowns, token totals, and reproducibility metadata.
- Register Integration: Populates the evaluation_report block in eval.yaml for register entries and commits report_config.yaml, report.md, and results/ alongside the eval.
- Use Case: A contributor adding a new benchmark to inspect_evals asks the agent to run the evaluation report workflow; it runs a 5-sample pilot, recommends three models across providers with a cost estimate, then generates and splices the final report into the README.
Quick Start
Ask the agent to make an evaluation report for a specific eval, for example: run the Make An Evaluation Report workflow for the gpqa eval.
