skills.rest
QingHuan-6
QingHuan-6COMMUNITY
Explore all skills by this author →
GitHub

llm-serving-auto-benchmark

Benchmark SGLang, vLLM, and TensorRT-LLM serving to produce Markdown/CSV summaries.

Updated May 30, 2026
Search Tags:#automation#validation#llm#benchmark#serving#cross-framework
One-click install
npx skills add https://github.com/QingHuan-6/SglangGen --skill llm-serving-auto-benchmark-qinghuan-6
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-serving-auto-benchmark
Source: https://github.com/QingHuan-6/SglangGen/tree/main/.claude/skills/llm-serving-auto-benchmark
Command: npx skills add https://github.com/QingHuan-6/SglangGen --skill llm-serving-auto-benchmark-qinghuan-6

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires PyYAML, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Benchmark cross-framework LLM serving to identify the best deployment commands across SGLang, vLLM, and TensorRT-LLM for a consistent workload set.

Core Features & Use Cases

  • Playbooks plus a config+validator toolchain that render framework-specific server and benchmark commands from cookbook configs.
  • Produces per-framework best commands, SLA pass/fail, and cross-framework comparisons, with artifact references and logs for audit.
  • Use cases include comparing model deployments under the same workload, validating flags against captured help outputs, and exporting Markdown/CSV summaries for decision making.

Quick Start

Run the validator and benchmark using cookbook configs to generate per-framework best commands and a cross-framework summary.

Frequently Asked Questions about llm-serving-auto-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark LLM serving performance across SGLang, vLLM, and TensorRT-LLM?▼

To benchmark LLM serving across SGLang, vLLM, and TensorRT-LLM, you can use a config and validator toolchain that renders framework-specific commands from cookbook configs. It validates flags against help outputs and runs a consistent workload set to identify the best deployment commands.

What is the best way to compare vLLM and TensorRT-LLM deployment commands for the same workload?▼

The best way to compare vLLM and TensorRT-LLM deployment commands is applying a cross-framework benchmark to a consistent workload set. This process validates configuration flags, renders framework-specific commands, and produces a Markdown or CSV summary for cross-framework comparison and decision making.

Does this cross-framework benchmark workflow support exporting results for decision making?▼

Yes, the cross-framework benchmark workflow exports results by producing Markdown and CSV summaries. These summaries include per-framework best commands, SLA pass or fail status, and cross-framework comparisons, along with artifact references and logs for audit.

Can I validate SGLang configuration flags before running the LLM serving benchmark?▼

Yes, you can validate SGLang configuration flags before running the LLM serving benchmark. The workflow includes a validation step that checks flags against captured help outputs to ensure the rendered server and benchmark commands are correctly configured.

Do I need PyYAML to run the cross-framework LLM deployment benchmark?▼

Yes, you need PyYAML installed to run the cross-framework LLM deployment benchmark. PyYAML is a required dependency for processing the cookbook configs used to render framework-specific server and benchmark commands.

Why should I use a cross-framework LLM benchmark instead of testing frameworks individually?▼

You should use a cross-framework LLM benchmark to identify the best deployment commands across SGLang, vLLM, and TensorRT-LLM for a consistent workload set. Testing individually lacks standardized SLA pass or fail comparisons and unified Markdown or CSV summaries for audit.

Related Skills

Explore More Software Engineering Skills→
hey-konghey-kong

llm-serving-auto-benchmark

Generate and validate deployment commands for comparing LLM serving frameworks.

Community
Advanced
gyy0592gyy0592
7

gen-report-detailed

Generate detailed 13-section markdown experiment reports from flame_moonshot training logs and metrics.

Community
Advanced
dyyounggdyyoungg

llm-torch-profiler-analysis

Analyze LLM torch-profiler traces to identify kernel, overlap, and fusion patterns.

Community
Advanced
annealing-inversionannealing-inversion

llm-serving-auto-benchmark

Benchmark LLM serving frameworks to identify the best deployment command.

Community
Advanced