skills.rest
air-gapped
air-gappedOFFICIAL
Explore all skills by this author →
GitHub

vllm-benchmarking

Benchmark production vLLM deployments for latency, throughput, and SLO compliance.

5|1|Updated Apr 19, 2026
Search Tags:#benchmarking#latency#throughput#benchmarks#slo#vllm#air-gapped
One-click install
npx skills add https://github.com/air-gapped/skills --skill vllm-benchmarking
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: vllm-benchmarking
Source: https://github.com/air-gapped/skills/tree/main/.claude/skills/vllm-benchmarking
Command: npx skills add https://github.com/air-gapped/skills --skill vllm-benchmarking

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

Benchmark production vLLM deployments to measure latency, throughput, and SLO compliance under realistic load. It applies to air-gapped environments and a variety of bench scenarios, including health checks, change comparisons, and SLO validation across multiple subcommands. It supports structured JSON output, warmup controls, percentile metrics, goodput budgets, and repeatable rate sweeps for reproducible performance analysis.

Core Features & Use Cases

  • Provides guidance and tooling for benchmarking vLLM deployments across serve, sweep, startup, latency, and throughput subcommands.
  • Demonstrates health-check, A/B change comparison, and SLO-constrained measurements with repeatable, auditable workflows.
  • Documents air-gapped patterns (mirrors, ModelScope substitutions, offline caches) and a complete dataset catalog for production-like workloads.

Quick Start

Run a health-check benchmark against a running vLLM deployment to generate baseline latency and throughput numbers.

Frequently Asked Questions about vllm-benchmarking

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark vLLM latency and throughput in production?▼

To benchmark vLLM latency and throughput in production, run health-check, startup, and latency subcommands to generate baseline performance metrics and structured JSON output under realistic load.

What is goodput budget validation for vLLM SLO compliance?▼

Goodput budget validation for vLLM SLO compliance measures whether your deployment meets service level objectives under load by applying goodput budgets and percentile metrics to ensure reproducible performance analysis.

Can I run vLLM benchmarks in an air-gapped environment?▼

Yes, you can run vLLM benchmarks in an air-gapped environment by utilizing offline caches, ModelScope substitutions, and mirrors to ensure repeatable tests without requiring external internet access.

How do I perform A/B change comparisons for vLLM deployments?▼

To perform A/B change comparisons for vLLM deployments, use the serve and sweep subcommands to execute repeatable rate sweeps, comparing baseline metrics against changes to validate performance regressions.

Do I need warmup controls to get accurate vLLM benchmark metrics?▼

Yes, you need warmup controls to get accurate vLLM benchmark metrics because they stabilize the environment before measurement, ensuring your latency, throughput, and SLO compliance data is reproducible.

Related Skills

Explore More Software Engineering Skills→
air-gappedair-gapped
5

vllm-configuration

Configure and verify vLLM deployments in air-gapped environments.

Official
Advanced
jacob-balslevjacob-balslev
1

performance-testing

Run controlled load tests measuring latency, throughput, error rate, and saturation against SLO thresholds.

Community
Intermediate
QingHuan-6QingHuan-6

llm-serving-auto-benchmark

Benchmark SGLang, vLLM, and TensorRT-LLM serving to produce Markdown/CSV summaries.

Community
Advanced
cosmixcosmix
53

performance-testing

Run load tests with k6, Locust, or Artillery and report latency, throughput, and error rate.

Community
Advanced