benchmark-survey

What problem does it solve?

This Skill automates the discovery and structured evaluation of benchmarks relevant to a project's formalized problem class, turning scattered literature and leaderboards into a prioritized set of candidate benchmarks and a written proposal for researcher review.

Core Features & Use Cases

• Problem extraction: Summarizes the formalization's observation/action/task vocabulary to build search queries for relevant benchmarks.
• Literature querying & extraction: Searches Papers With Code, alpha_review APIs, and surveys; reads benchmark papers and extracts task scope, metrics, baselines, install recipes, hardware needs, and community usage.
• Ranking & recommendation: Scores candidates on coverage, non-saturation, community adoption, and install effort, then flags recommendations and produces a benchmark_proposal.md for human confirmation.
• Use Case: Use during FORMALIZE to surface in-scope benchmarks before writing benchmarks.md or during APPROACH to check competitor evaluation coverage.

Quick Start

Run the benchmark-survey skill on your project directory with project.md and formalization.md to generate benchmark_proposal.md for human review.