terminal-bench-loop

Drives one Terminal-Bench task through bounded smoke, diagnosis, and board-gated fix iterations.

Updated Mar 26, 2026
One-click install
npx skills add https://github.com/nabitllc/todero --skill terminal-bench-loop-nabitllc
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: terminal-bench-loop
Source: https://github.com/nabitllc/todero/tree/main/.agents/skills/terminal-bench-loop
Command: npx skills add https://github.com/nabitllc/todero --skill terminal-bench-loop-nabitllc

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running a Terminal-Bench task repeatedly against a Todero worktree often stalls in ambiguous review states, unbounded retries, or fixes applied without approval. This Skill enforces a bounded loop with explicit issue topology, typed waiters, and board confirmation so every iteration ends in a known state with a named next-action owner. ## Core Features & Use Cases - Bounded iteration loop: Runs a single Terminal-Bench task as a smoke with an iteration budget, wall-clock caps, and explicit stop rules (pass, board rejection, budget exhausted, or named blocker). - Structured diagnosis: Applies a diagnose-why-work-stopped pattern to classify failures as task/model, Todero product, harness/setup, verifier/infrastructure, security, or unclear, with evidence quoted from run ids and status transitions. - Board-gated fixes: Requires a request_confirmation interaction on the iteration child before any implementation child is created, keeping product fixes auditable and approved. - Worktree continuity: Pins the benchmark command to an isolated Todero App worktree via PAPERCLIPAI_CMD and inheritExecutionWorkspaceFromIssueId so fixes and reruns share one workspace. - Use Case: An operator is asked to drive terminal-bench/fix-git until it passes; the Skill opens a loop parent issue, runs bounded smokes, diagnoses each stop, proposes product rules for board approval, and closes with QA and CTO review. ## Quick Start Ask the agent to loop the terminal-bench/fix-git task through Todero with an iteration budget of three until the smoke passes.

Frequently Asked Questions about terminal-bench-loop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Terminal-Bench task in a loop until it passes?

Open a top-level loop issue with the task name, iteration budget, benchmark command, and worktree pointer, then create one iteration child per run. Each iteration runs a bounded smoke, records a diagnosis, and stops on pass, board rejection, budget exhaustion, or a named blocker.

How do I diagnose why a Terminal-Bench run stopped?

Walk the Todero issue tree the smoke produced and find the exact issue and status that stopped progress, quoting run ids and status transitions. Classify the failure as task/model, Todero product, harness/setup, verifier/infrastructure, security, or unclear before proposing any fix.

Can this skill submit comparable Terminal-Bench benchmark results?

No. The loop produces smoke runs that are explicitly non-comparable by design. Comparable or full-suite benchmark submissions must be handed off to BenchmarkQualityManager and BenchmarkForensics instead of relabeling a smoke run.

When should I not use the Terminal-Bench loop skill?

Do not use it to build or change the todero-bench harness itself, to submit ranked benchmark results, or for ordinary Todero product bugs unrelated to a loop. Those cases follow normal engineering or investigation flows.

Why does a Harbor run create BEN-1 with no heartbeat agents?

That happens when the dispatch runner config is omitted, leaving the benchmark issue unassigned with zero heartbeat-enabled agents. Rerun with the full recorded command block including PAPERCLIP_HARBOR_RUNNER_CONFIG, assignee, heartbeat strategy, and agent adapter.

Can the loop apply product fixes without board approval?

No. Every product fix requires a request_confirmation interaction on the iteration child targeting the latest plan revision. Implementation, QA, and rerun children are only created after the board accepts the proposal.