terminal-bench-task-reviewer

You validate a Terminal-Bench task directory by checking for required components like task.yaml, Dockerfile, solution.sh, and tests, ensuring proper structure, pinned dependencies, and visible frontmatter for compliance.

A valid SKILL.md for a Terminal-Bench task requires mandatory name and description fields in the frontmatter, plus a body providing actionable guidance to pass quality audits.

Your Terminal-Bench task fails compliance if it has missing or misnamed components like Dockerfile or task.yaml, lacks visible frontmatter, uses unpinned dependencies, or has insufficient test coverage.

You check Dockerfile compliance by running a task review audit that verifies proper structure, pinned dependencies, and cross-references timeout settings and test coverage against Terminal-Bench best practices.

Yes, you can apply this audit to review or validate a Terminal-Bench task before canary testing by generating a concise compliance report that highlights critical findings and suggested fixes.

The best way to ensure task.yaml correctness is to perform a quick quality audit cross-referencing instructions, timeout settings, and test coverage against Terminal-Bench standards before submission.