tbench

Run Terminal-Bench benchmarks for Unix agents by setting the TB_ENV variable to target a local Docker environment, then execute the suite to automatically collect per-task outcomes and logs under the runs/ directory for performance analysis.

Yes, run specific Terminal-Bench tasks by defining the TB_TASK_NAMES environment variable, allowing targeted benchmarking and failure analysis for specific scenarios like chess moves or maze navigation instead of executing the entire suite.

Configure Unix agent benchmarking concurrency and timeouts by setting the TB_CONCURRENCY and TB_TIMEOUT environment variables, which control parallel execution limits and maximum allowed runtime per task during the Terminal-Bench evaluation.

Terminal-Bench supports cloud sandbox environments for agent evaluation by configuring the TB_ENV variable to target Daytona cloud sandboxes, enabling remote execution of benchmark suites with configurable datasets and concurrency settings.

Analyze agent benchmark failures by reviewing the per-task outcomes and logs automatically collected under the runs/ directory after Terminal-Bench execution, then integrating with Harbor for comprehensive evaluation and leaderboard submissions.

Harbor integration is used for evaluation and leaderboard submissions after Terminal-Bench runs, working alongside the automatically collected run results and logs to measure Unix-based agent performance and identify bottlenecks.