What problem does it solve?
Improving an AI agent plugin requires knowing which tools and skills actually fail on real questions, but manual testing is slow and fixes often overfit to a single benchmark's answer key. This Skill provides a 5-step feedback loop (run, analyze, diagnose, fix, retest) that continuously measures and improves ToolUniverse tools, skills, and plugin quality.
Core Features & Use Cases
- Orchestrated benchmark runs: Execute lab-bench, BixBench, or custom question sets in plugin-only, baseline-only, or comparison modes with a single script that builds, runs, analyzes, and extracts failures.
- Failure diagnosis and routing: Classify failures (timeout, wrong_answer, tool_error, api_key_missing), map question categories to responsible skills, and route each fix to the correct devtu skill instead of manual patching.
- Multi-strategy grading: Grade answers with exact, multiple-choice, range, normalized, numeric-proximity, synonym, and LLM-verifier strategies, including Unicode scientific notation handling.
- Anti-memorization guard: Scan skill files for benchmark names, capsule UUIDs, question IDs, and known ground-truth values to prevent overfitting conventions to a dataset's answer key.
- Use Case: After adding a new ToolUniverse tool, run the harness on 20 BixBench questions, review the diagnose log to see which skill underperforms, invoke the recommended devtu skill to fix it, then retest only the failed questions to confirm the improvement.
Quick Start
Run the benchmark harness on 20 BixBench questions with seed 42, then show me the diagnosed failures and which devtu skills to invoke for fixes.
