tool-eval-bench
CommunityBenchmark LLM tool-calling quality across various scenarios
AuthorSeraphimSerapis
Version1.0.0
Installs0
System Documentation
What problem does it solve?
Evaluates the ability of LLMs to call tools correctly and efficiently, identifying issues with tool selection, parameter precision, error handling, and more.
Core Features & Use Cases
- 69 Scenarios: Tests LLMs across 15 categories, including tool selection, parameter precision, multi-step chains, error recovery, and more.
- Throughput Benchmark: Measures prefill and token generation speed.
- Pluggable Accuracy Benchmarks: Integrates external benchmarks like GSM8K, MMLU, and IFEval for accuracy evaluation.
- Use Case: Use this tool to assess the tool-calling quality of LLMs in various agentic workflows, ensuring they can handle tasks like data retrieval, decision-making, and task orchestration.
Quick Start
Run the benchmark with the following command:
tool-eval-bench --short
Dependency Matrix
Required Modules
httpxpytestruffyaml
Components
scriptsreferencesassets
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: tool-eval-bench Download link: https://github.com/SeraphimSerapis/tool-eval-bench/archive/main.zip#tool-eval-bench Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.