What problem does it solve?
Evaluations in the inspect_evals repository may make false claims about their data provenance, use misleading names, contain samples that cannot be both passed and failed, or score proxies instead of ground truth. This Skill provides a structured, read-only review process that determines whether an evaluation actually measures what it claims to measure.
Core Features & Use Cases
- Claims Coherence Verification: Extracts every verifiable claim from the README, eval.yaml, docstrings, and dataset, then checks each against the actual code and data, flagging fabricated provenance or false mechanism claims.
- Name, Dataset, and Scoring Validity Checks: Assesses whether the eval name matches its real scope, whether models can both succeed and fail given available tools and sandbox affordances, and whether the scorer measures ground truth rather than weak proxies like substring matching.
- Structured Report Generation: Produces a VALIDITY_REPORT.md with an overall rating (Valid, Valid with Minor Issues, Validity Concerns, or Significant Validity Issues), severity-tagged findings, and prioritized recommendations.
- Use Case: Before merging a community-contributed evaluation into inspect_evals, run this review to confirm its HuggingFace dataset claims are real, its sandbox tasks are solvable, and its LLM-judge scorer verifies actual task completion.
Quick Start
Ask the AI to review the validity of the evaluation in src/inspect_evals/<eval_name> and generate a validity report.
