skills.rest
SeraphimSerapisSeraphimSerapisCommunity·1 Agent Skills Included

tool-eval-bench

Benchmark LLM tool-calling quality across 88 deterministic scenarios

Evaluates how well language models select tools, pass parameters, chain steps, and handle errors across 88 deterministic scenarios. Scores each run as pass, partial, or fail with full conversation traces saved to Markdown and SQLite. Measures throughput, long-context retrieval, and accuracy against vLLM, SGLang, LiteLLM, llama.cpp, and Gemini endpoints. Replaces manual spot-checking with reproducible, comparable benchmark reports.
285|33
↗
npx skills add SeraphimSerapis/tool-eval-bench --all -g -y
Available:

Gives your AI agent the project's architecture rules, scenario and plugin conventions, storage requirements, and quality checks to follow when extending or running this benchmark.

All Skills in This Repository (1)

Pure Emerald Level Indicators
📦 In Repo
SeraphimSerapisSeraphimSerapis

tool-eval-bench

Evaluate LLM tool-calling quality across 69 deterministic scenarios.

Community
Advanced

Frequently Asked Questions

FAQPage Schema
How to install tool-eval-bench?▼

Run `npx skills add SeraphimSerapis/tool-eval-bench --all -g -y` in your terminal to install everything globally.

What does tool-eval-bench measure?▼

It measures tool-calling quality: whether a model picks the right tool, passes correct parameters, chains steps, and respects safety boundaries across 88 scenarios. It also benchmarks throughput, long-context retrieval, and accuracy via GSM8K, MMLU, and IFEval.

Which inference servers does it support?▼

It works with any OpenAI-compatible endpoint, including vLLM, SGLang, LiteLLM, llama.cpp, and Ollama, plus native Gemini support.

How do I compare two benchmark runs?▼

Every run is saved to SQLite and a Markdown report, so you can run `tool-eval-bench compare A B` or view ranked results with `tool-eval-bench leaderboard`.

Can I use it without writing code?▼

Yes. The command-line interface handles everything; just point it at your server's URL and run `tool-eval-bench run --short` for a quick evaluation.

Related Repositories in Software Engineering

View All in Software Engineering→
openclawopenclaw
388.4k

openclaw

Run a personal AI assistant across your devices and chat apps

Official📦 120 Skills
obraobra
280.0k

superpowers

Gives coding agents a disciplined workflow from idea to merged code

Community📦 15 Skills
facebookfacebook
248.4k

react

AI agent skills for building, testing, and porting React core

Official📦 14 Skills