skills.rest
brazil-bench
brazil-benchOFFICIAL
Explore all skills by this author →
GitHub

evaluate-attempt

Evaluates Brazil-bench attempts and generates structured reports on conformance, tests and timeline for Python/Swift-iOS projects.

11|Updated Nov 24, 2025
Search Tags:#automation#metrics#Python#benchmark#Swift#evaluate#conformance
One-click install
npx skills add https://github.com/brazil-bench/pourpoise --skill evaluate-attempt
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: evaluate-attempt
Source: https://github.com/brazil-bench/pourpoise/tree/main/skills/evaluate-attempt
Command: npx skills add https://github.com/brazil-bench/pourpoise --skill evaluate-attempt

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This SOP evaluates a completed brazil-bench attempt against the spec.md requirements, capturing metrics for comparison across orchestration patterns. It ensures consistent evaluation across Python and Swift/iOS projects.

Core Features & Use Cases

  • Conformance Metrics: automatically verify alignment with the benchmark spec and produce a structured report.
  • Multi-language Support: supports Python and Swift/iOS implementations, adapting checks to the language and tooling.
  • Use Case: after an attempt is completed, run this Skill to generate a leaderboard-ready report detailing conformance, test outcomes, and development timeline.

Quick Start

Run the evaluate-attempt skill against a repository and specify an output directory, for example: evaluate-attempt --attempt_repo=attempt-3 --output_dir=./results

Frequently Asked Questions about evaluate-attempt

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I evaluate a benchmark attempt against spec.md requirements?▼

This Skill validates benchmark attempts by cloning the repository, detecting spec.md modifications, and producing a structured evaluation report with conformance metrics, test results, and an autonomous development duration estimate.

Can I evaluate Python and Swift iOS implementations using the same benchmark metrics?▼

Yes, you can evaluate both Python and Swift/iOS implementations. The Skill automatically identifies the primary implementation language and adapts its conformance checks and evaluation metrics to the specific tooling of the project.

How do I generate a leaderboard-ready report for benchmark conformance?▼

You generate a leaderboard-ready report by running the evaluation against a completed attempt repository. It automatically captures conformance metrics, test outcomes, and autonomous duration estimates for comparison across orchestration patterns.

Does the benchmark evaluation detect modifications to the original spec.md?▼

Yes, the benchmark evaluation detects spec.md modifications. It clones the attempt repository into a local reviews directory, compares the attempt's spec.md against the benchmark-template, and flags any changes in the final report.

What metrics are included in a benchmark attempt evaluation report?▼

The evaluation report includes conformance metrics, test results, and the development timeline. It specifically provides an autonomous duration estimate alongside the structured conformance data to ensure consistent comparison across orchestration patterns.

Related Skills

Explore More Data & Analytics Skills→
pradeepmoulipradeepmouli
2

speckit-baseline

Generate baseline-spec.md and current-state.md from repository analysis.

Community
Advanced
brazil-benchbrazil-bench
11

compare-attempts

Compare brazil-bench evaluation reports and generate a ranked Markdown leaderboard.

Official
Advanced
no-problem-devno-problem-dev
5

ios-test-runner

Execute iOS unit tests and Swift package tests via CLI.

Official
Intermediate
Zhow01Zhow01
52

llm-evaluation

Assess and benchmark LLM outputs across automated metrics, human judgments, and safety checks.

Community
Advanced