bench-task-judge

To evaluate benchmark task difficulty and fairness, you analyze task statistics, run static rubric checks, execute mutation harness tests, and replay saved solutions to ensure the task meets inclusion criteria.

Benchmark task fairness analysis checks if a verifier is unbiased by running static rubric checks and mutation harness tests against the task to ensure the verification process is equitable for all models being assessed.

Yes, you need access to the diolog-swe-bench repository and its associated tools for task execution and analysis. You must run the evaluation from the repository root and provide the specific task ID.

You can run difficulty assessments on any benchmark task by providing its task ID. The evaluation determines whether a task is challenging enough for given models by analyzing task statistics and replaying saved solutions.

The best way to check if a benchmark task is too easy is to run a comprehensive evaluation that analyzes task statistics, applies mutation harness tests, and replays saved solutions to measure the challenge level for given models.

Benchmark task evaluation requires mutation harness tests and solution replays to statically verify rubric correctness and dynamically confirm that task difficulty and verifier fairness hold against actual execution results.