compare-attempts

Benchmarking multiple evaluation reports involves analyzing markdown files compiled by the Evaluate Attempt SOP. The tool consolidates these reports, normalizes data against a canonical 16-requirement denominator, and outputs a detailed comparison summary with per-attempt trend insights and patterns.

Composite scores for benchmarking attempts are calculated using four key metrics: spec compliance, effective test coverage, code quality, and efficiency. The tool extracts these metrics from evaluation reports to rank attempts accurately on the leaderboard.

Yes, you can compare evaluation reports across multiple attempts without manual data consolidation. The tool automatically enforces canonical 16-requirement denominators, extracts relevant metrics, and consolidates results into a unified Markdown leaderboard and detailed analytics summary.

To set up the results directory for benchmarking evaluation reports, set the results_dir parameter to the folder containing your markdown reports. Execute the comparison workflow to output the generated Markdown leaderboard to your specified output_file.

The canonical 16-requirement denominator is a standardized validation baseline used in benchmark evaluation comparison. The tool enforces this denominator during automated validation to normalize spec compliance and metrics extraction across all compared attempts.