bench-corpus-review

Automating corpus review involves re-running quality checks across all tasks to generate per-task verdicts. This process coordinates with task judges for binary checks and mechanically diffs results against previous reviews to track benchmark health.

Auditing benchmark tasks requires running a whole-corpus review to assess every task. You assemble documentation of per-task verdicts and summary counts, then diff the current results against the previous review to identify changes and trends.

Corpus diffing works by mechanically comparing the current whole-corpus review against the previous one. This comparison identifies changes and trends in task verdicts, ensuring ongoing quality control after new tasks are added or modifications are made.

Yes, you can use bench-task-judge for binary checks and decision tables during a whole-corpus review. The review process coordinates with this task judge to automate quality control and assemble per-task verdicts for the benchmark.

You need to run a whole-corpus quality review when auditing the benchmark, after new tasks have been added to the corpus, or on a scheduled periodic basis. This ensures ongoing quality control and verifies benchmark health.

Yes, you need to run the corpus review skill from the diolog-swe-bench repository root. Running from this root directory allows the automated review to properly regenerate the corpus-wide task documentation and coordinate with task judges.