bench-task-judge
OfficialEvaluate benchmark tasks for difficulty and fairness.
Software Engineering#benchmark#model assessment#task evaluation#fairness analysis#difficulty assessment
AuthorDiologIR
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill provides a comprehensive evaluation of benchmark tasks, ensuring they are difficult enough and fair to the verifiers, helping maintain the quality and relevance of the benchmark.
Core Features & Use Cases
- Difficulty Assessment: Determines whether a task is challenging enough for the given models.
- Fairness Analysis: Checks if the verifier is fair and unbiased.
- Task Review: Offers a detailed review with specific changes and recommendations.
- Use Case: Before adding a new task to the benchmark, use this Skill to ensure it meets the necessary criteria for inclusion.
Quick Start
To evaluate a task, run the bench-task-judge skill from the diolog-swe-bench repo root and provide the task ID.
Dependency Matrix
Required Modules
None requiredComponents
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: bench-task-judge Download link: https://github.com/DiologIR/diolog-plugins/archive/main.zip#bench-task-judge Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.