skills.rest
hey-kong
hey-kongCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

llm-serving-auto-benchmark

Generate and validate deployment commands for comparing LLM serving frameworks.

Updated Jun 29, 2026
Search Tags:#automation#validation#benchmark#cross-framework#cookbook#llm-serving
One-click install
npx skills add https://github.com/hey-kong/sglang --skill llm-serving-auto-benchmark-hey-kong
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-serving-auto-benchmark
Source: https://github.com/hey-kong/sglang/tree/main/.claude/skills/llm-serving-auto-benchmark
Command: npx skills add https://github.com/hey-kong/sglang --skill llm-serving-auto-benchmark-hey-kong

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Framework-agnostic benchmark orchestration to help users identify the best deployment commands for comparing LLM serving frameworks across models and workloads, ensuring fair and reproducible results.

Core Features & Use Cases

  • Config-driven cross-framework benchmarking: Compare SGLang, vLLM, and TensorRT-LLM using the same workload, GPU budget, and SLA targets.
  • Cookbook-driven command generation: Translate high-level cookbook configs into framework-native candidate commands with validated help outputs.
  • Normalized result schema and SLA ranking: Produce a unified JSONL result set and markdown summary to rank winners by SLA compliance and throughput.

Quick Start

Run a cookbook-driven cross-framework benchmark by supplying your config files and let the skill render and validate candidate commands, then review the resulting summary.

Frequently Asked Questions about llm-serving-auto-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark LLM serving frameworks like SGLang, vLLM, and TensorRT-LLM fairly?▼

Cross-framework LLM serving benchmarks compare SGLang, vLLM, and TensorRT-LLM by applying the same workload, GPU budget, and latency SLA. This validates cookbook configs, generates framework commands, and normalizes results for SLA-based ranking.

What is the best way to generate deployment commands for LLM serving benchmarks?▼

Generating deployment commands for LLM serving benchmarks uses cookbook-driven configs to translate high-level settings into framework-native candidate commands. It validates commands against help outputs to ensure proper execution before running the benchmark.

How do I rank LLM serving frameworks by SLA compliance and throughput?▼

Ranking LLM serving frameworks by SLA compliance and throughput requires normalized result schemas. The benchmark produces a unified JSONL result set and markdown summary to rank winners based on SLA compliance and throughput metrics.

Do I need pyyaml to run cross-framework LLM serving benchmarks?▼

Yes, you need pyyaml to run cross-framework LLM serving benchmarks. It is the required dependency for parsing the cookbook configs that drive the config-driven benchmarking process across different frameworks.

Can I use YAML configs to orchestrate LLM serving benchmarks across multiple models?▼

Yes, you can use YAML cookbook configs to orchestrate LLM serving benchmarks across multiple models and workloads. The config-driven approach ensures fair and reproducible results by maintaining consistent GPU budgets and SLA targets.

Related Skills

Explore More Software Engineering Skills→
annealing-inversionannealing-inversion

llm-serving-auto-benchmark

Benchmark LLM serving frameworks to identify the best deployment command.

Community
Advanced
dyyounggdyyoungg

llm-serving-auto-benchmark

Run unified benchmarks across SGLang, vLLM, and TensorRT-LLM serving frameworks.

Community
Intermediate
QingHuan-6QingHuan-6

llm-serving-auto-benchmark

Benchmark SGLang, vLLM, and TensorRT-LLM serving to produce Markdown/CSV summaries.

Community
Advanced
BBufBBuf
721

llm-serving-auto-benchmark

Benchmark LLM serving frameworks across SGLang, vLLM, and TensorRT-LLM.

Community
Advanced