Evaluate LLM tool-calling quality across 69 deterministic scenarios.
npx skills add https://github.com/SeraphimSerapis/tool-eval-bench --skill tool-eval-bench
Please help me install this Agent Skill. Skill: tool-eval-bench Source: https://github.com/SeraphimSerapis/tool-eval-bench/tree/main Command: npx skills add https://github.com/SeraphimSerapis/tool-eval-bench --skill tool-eval-bench
Evaluates the ability of LLMs to call tools correctly and efficiently, identifying issues with tool selection, parameter precision, error handling, and more.
Run the benchmark with the following command:
tool-eval-bench --short
High-intent search queries and answers about installing and using this skill.
You can benchmark LLM tool-calling quality by running 69 deterministic scenarios across 15 categories, evaluating tool selection, parameter handling, multi-step chains, and error recovery. Use the command `tool-eval-bench --short` to quickly start the evaluation.
LLM tool-calling evaluation measures performance in tool selection, parameter precision, error handling, and efficiency. It places specific emphasis on multi-step task chaining and context management to ensure models handle complex orchestration.
Yes, you can evaluate model accuracy alongside throughput benchmarks. The framework integrates pluggable external benchmarks like GSM8K, MMLU, and IFEval for accuracy evaluation while measuring prefill and token generation speed.
Yes, you need an OpenAI-compatible endpoint and model access to test LLM tool selection. The evaluation framework relies on these endpoints to execute scenarios and assess how well the model retrieves data and orchestrates tasks.
Deterministic scenarios provide controlled evaluation across 15 categories but may not capture the full unpredictability of real-world agentic workflows. They focus specifically on tool selection, parameter precision, and multi-step error recovery.
QianWen-AIEnable OpenAI-compatible chat, text generation, and tool calling for Qwen models.
SakshamKandelBenchmark Codex, GPT, and Gemini with the same prompt for latency, tokens, cost, and quality.
kartikfedRun Black, Ruff, Mypy, and Pytest checks on Python code.
informatico-madridEvaluate LLMs across 100+ benchmarks using Docker, Slurm, or cloud backends.