skills.rest
Cloud-Byte-Consulting
Cloud-Byte-ConsultingOFFICIAL
Explore all skills by this author →
GitHub

inference-benchmark-runner

Benchmarks LLM inference latency, throughput, GPU utilization, and cost on Kubernetes.

Updated Jul 16, 2026
Search Tags:#llm#benchmarking#kubernetes#load testing#gpu#inference#ttft
One-click install
npx skills add https://github.com/Cloud-Byte-Consulting/plugins --skill inference-benchmark-runner-cloud-byte-consulting
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: inference-benchmark-runner
Source: https://github.com/Cloud-Byte-Consulting/plugins/tree/main/inference-testing/skills/inference-benchmark-runner
Command: npx skills add https://github.com/Cloud-Byte-Consulting/plugins --skill inference-benchmark-runner-cloud-byte-consulting

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Teams promoting LLM checkpoints often lack defensible performance numbers: they conflate quality with speed, average cold and warm starts, quote concurrency-1 costs, and report GPU utilization as if it were application performance. This Skill produces reproducible benchmark comparisons of model inference on GPU Kubernetes so release decisions rest on measured latency, throughput, and cost. ## Core Features & Use Cases - Concurrency sweep benchmarking: Steps load levels with k6 or Locust to find the saturation knee, recording TTFT, inter-token latency, tokens/sec, and p50/p95/p99 at each level. - DCGM GPU correlation: Scrapes NVIDIA DCGM metrics into Prometheus to diagnose whether a model is compute-bound, memory-bandwidth-bound, or VRAM-limited. - Cost modeling: Derives cost-per-1k-tokens and cost-per-request at the operating concurrency, including idle-endpoint and batch economics. - Use Case: Compare a 7B and a 70B checkpoint under the same vLLM runtime and workload profile, then hand the SLO pass/fail to the release gate and the saturation knee to the rollout strategist for autoscaling policy. ## Quick Start Use the inference-benchmark-runner skill to benchmark my two candidate models on the GPU cluster and produce a comparison report with TTFT, throughput, and cost per 1,000 tokens.

Frequently Asked Questions about inference-benchmark-runner

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark LLM inference latency and throughput on Kubernetes?▼

Run a concurrency sweep with a load tool like k6 or Locust against the serving endpoint, stepping concurrency while holding input and output lengths fixed. Record TTFT, inter-token latency, tokens/sec, and p95/p99 at each level to find the saturation knee, which is the usable capacity.

What is TTFT and why does it matter for LLM serving?▼

TTFT (time to first token) is the latency from request to the first streamed token, and it is the number users actually feel. Interactive UIs notice latency around 250ms, so p95 TTFT in the hundreds-of-milliseconds range is usually the binding SLO for interactive workloads.

How do I measure GPU utilization during LLM inference benchmarks?▼

Scrape GPU telemetry with the NVIDIA DCGM exporter into Prometheus and correlate SM utilization, memory-bandwidth utilization, VRAM, and power draw with request-level metrics. High SM occupancy with low tokens/sec means you are memory-bound or batching poorly, not well utilized.

How do I calculate cost per 1,000 tokens for self-hosted models?▼

Divide the GPU hourly cost by system tokens/sec times 3.6, computed at the operating concurrency (the saturation knee), not at concurrency 1. Single-request cost figures ignore batching economics and overstate real cost several-fold.

Why should cold start and warm inference benchmarks be separated?▼

Cold starts include weight loading, CUDA initialization, and cache warm-up, which can take minutes for large models, while warm numbers reflect steady state. Averaging them understates steady-state speed and hides cold-start risk that drives readiness-probe timeouts and autoscaling headroom.

Does the serving runtime affect LLM benchmark results?▼

Yes, the same weights on the same GPU produce different numbers under vLLM, TGI, TensorRT-LLM, or a plain framework server due to differences in batching, KV-cache management, and quantization. Pin one runtime when comparing models, and record runtime, version, quantization, and tensor-parallel degree in every report.

Related Skills

Explore More Software Engineering Skills→
coreweavecoreweave
7

setup-sunk-gpu-monitoring

Set up GPU and Slurm metrics collection on a SUNK cluster.

Official
Advanced
Clyra-AIClyra-AI
12

gait-policy-test-rollout

Execute deterministic policy tests and rollout simulations from policy YAML and intent fixtures.

Official
Advanced
kreuzberg-devkreuzberg-dev
9.2k

benchmark-workflow

Run and diagnose Xberg extraction benchmarks, quality scoring, and ground-truth fixtures.

Official
Intermediate
masermediagroup-stackmasermediagroup-stack
1

model-serving

Deploy LLM and ML models for production inference with vLLM, BentoML, and streaming APIs.

Community
Advanced