skills.rest
jcrabapple
jcrabappleCOMMUNITY
Explore all skills by this author →
GitHub

llm-performance-benchmark

Find LLM tokens-per-second, latency, and throughput metrics from live tests or published benchmarks.

10|2|Updated May 6, 2026
Search Tags:#llm#latency#benchmark#throughput#tokens per second#ttft#provider comparison
One-click install
npx skills add https://github.com/jcrabapple/hermes-skills --skill llm-performance-benchmark-jcrabapple
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-performance-benchmark
Source: https://github.com/jcrabapple/hermes-skills/tree/main/llm-performance-benchmark
Command: npx skills add https://github.com/jcrabapple/hermes-skills --skill llm-performance-benchmark-jcrabapple

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? It answers questions about how fast a specific LLM actually runs by measuring tokens-per-second, TTFT, and throughput, or by locating trustworthy third-party benchmark data when direct API testing is not possible. ## Core Features & Use Cases - Live Benchmarking: Runs a timed generation against an available API key and computes TPS from completion tokens over elapsed time. - Third-Party Benchmark Research: Pivots to trusted aggregators like Artificial Analysis, DeepInfra, and LLM-Stats when direct measurement fails. - Provider Comparison: Highlights that the same model can vary 3x in speed across providers like Groq, Fireworks, or the official API. - Use Case: A user asks how fast Kimi K2.6 runs. The skill checks for an API key, falls back to published benchmarks, distinguishes it from earlier K2 variants, and reports median TPS and TTFT per provider with caveats. ## Quick Start Ask the agent to benchmark the tokens-per-second and latency of a specific model such as kimi-k2.6 across available providers.

Frequently Asked Questions about llm-performance-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure tokens per second for an LLM API?▼

Run a timed generation of roughly 400-500 tokens with your API key and compute TPS as completion tokens divided by elapsed time. If the key is masked in config files, stop after 2-3 attempts and use published benchmarks instead.

What are the best sites for LLM benchmark data?▼

Artificial Analysis tracks provider-level TPS, TTFT, and pricing. DeepInfra publishes provider shootouts, and LLM-Stats offers leaderboard rankings and quality scores. Cross-check these before trusting marketing numbers.

Why does the same model have different speeds on different providers?▼

Provider infrastructure varies significantly, so a model can run at 200 t/s on Groq but only 27 t/s on Novita. Always name the provider alongside the model when citing speed figures.

Why is TTFT so high for reasoning models?▼

Reasoning models stream thinking tokens before the actual answer, which inflates time to first token. Always clarify whether a cited TTFT figure includes reasoning time or only measures the answer stream.

What if my API key is redacted in the config file?▼

Do not spend more than 2-3 attempts extracting a masked key. Move immediately to web research on third-party benchmarks and provide the user a copy-paste script they can run with their own key.

Related Skills

Explore More Software Engineering Skills→
DalomeveDalomeve
3

blocker-min-input

Capture exact blocker text, attempts, and minimum unblock input from tool outputs.

Community
Intermediate
reaatechreaatech
2

Cost Optimization Skill

Route LLM requests to cost-effective models while meeting latency and quality targets.

Community
Advanced
BaGyun0107BaGyun0107

benchmark-models

Compare response quality, speed, and cost across Claude, GPT, and Gemini models.

Community
Intermediate
fgarofalo56fgarofalo56
6

localai

Run local AI models via a self-hosted OpenAI-compatible API.

Community
Advanced