terminal-bench-loop

Drives one Terminal-Bench task through a bounded Paperclip smoke, diagnosis, and fix loop.

Updated Jun 16, 2026
One-click install
npx skills add https://github.com/adamtpang/summon.company --skill terminal-bench-loop-adamtpang
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: terminal-bench-loop
Source: https://github.com/adamtpang/summon.company/tree/main/.agents/skills/terminal-bench-loop
Command: npx skills add https://github.com/adamtpang/summon.company --skill terminal-bench-loop-adamtpang

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running a Terminal-Bench task against Paperclip often stalls in ambiguous states: silent in_review issues, unbounded retries, fixes applied without approval, or smoke runs testing the wrong checkout. This Skill imposes a bounded, auditable loop with explicit issue topology, board-gated fixes, and worktree continuity so every iteration ends in a known state. ## Core Features & Use Cases - Bounded iteration loop: Runs a Terminal-Bench smoke with an iteration budget, wall-clock caps, and explicit stop rules (pass, board rejection, budget exhausted, or named blocker). - Board-gated product fixes: Diagnoses the exact stop point using the /diagnose-why-work-stopped pattern, then requires a request_confirmation approval before any implementation child issue is created. - Worktree continuity: Pins the benchmark command (PAPERCLIPAI_CMD) and dispatch runner config to one isolated Paperclip App worktree shared across all iterations via inheritExecutionWorkspaceFromIssueId. - Use Case: A user asks to drive terminal-bench/fix-git until it passes. The Skill opens a loop parent issue, runs bounded smokes, diagnoses failures, proposes product rules for board confirmation, and reruns against the same worktree until pass or budget exhaustion. ## Quick Start Ask the agent to run the Terminal-Bench task fix-git through a Paperclip smoke loop with an iteration budget of three and board approval required before any product fix.

Frequently Asked Questions about terminal-bench-loop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Terminal-Bench task in a loop until it passes?

Open a top-level loop issue with the task name, iteration budget, benchmark command, and dispatch config, then create one iteration child per run. Each iteration runs a bounded smoke, diagnoses the stop point, and either proposes a board-gated fix or stops with a named reason.

How do I diagnose why a Terminal-Bench run stopped in Paperclip?

Apply the /diagnose-why-work-stopped pattern to the smoke's issue subtree: walk each node, find the exact (issue, status) that stopped progress, and classify it as needing human intervention, agent-actionable, or already covered. Record the failure taxonomy bucket on the iteration child.

Can this skill submit comparable Terminal-Bench benchmark results?

No. This skill produces smoke runs that are explicitly non-comparable by design. For full-suite or comparable benchmark submissions, escalate to BenchmarkQualityManager and BenchmarkForensics instead of relabeling a smoke run.

Why must the benchmark run against an isolated Paperclip worktree?

The loop tests the same worktree where accepted fixes land, so results reflect the code under change. PAPERCLIPAI_CMD is pinned to that worktree's CLI entrypoint, and later iterations inherit the workspace via inheritExecutionWorkspaceFromIssueId.

When should I not use the Terminal-Bench loop skill?

Do not use it to build or change paperclip-bench itself, to submit ranked benchmark results, or for ordinary Paperclip bugs not surfaced by a loop. Those cases use normal engineering flow, BenchmarkQualityManager, or standard investigation respectively.

What happens when the iteration budget is reached without a pass?

The loop must stop: the loop issue moves to cancelled with a summary of run history, or to in_review if the user must decide whether to extend the budget. Starting another iteration silently is explicitly forbidden.