skills.rest
BBuf
BBufCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

vllm-sota-humanize-loop

Runs an autonomous Humanize RLCR loop that benchmarks, profiles, and patches vLLM until it matches the best framework.

783|67|Updated Apr 1, 2026
Search Tags:#benchmark#performance tuning#vllm#kernel optimization#LLM serving#torch profiler#rlcr loop
One-click install
npx skills add https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS --skill vllm-sota-humanize-loop
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: vllm-sota-humanize-loop
Source: https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS/tree/main/skills/vllm-sota-humanize-loop
Command: npx skills add https://github.com/BBuf/AI-Infra-Auto-Driven-SKILLS --skill vllm-sota-humanize-loop

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Closing the performance gap between vLLM and faster serving frameworks like SGLang or TensorRT-LLM normally requires manual benchmarking, profiler triage, and iterative code patching. This Skill automates that entire campaign as one recoverable, evidence-driven loop.

Core Features & Use Cases

  • Fixed fair benchmark gate: Runs a bounded cross-framework deployment search across vLLM, SGLang, and TensorRT-LLM with identical model, workload, precision, and SLA before any patching.
  • Humanize RLCR loop: Each round decides the remaining gap, runs torch-profiler and layer-pipeline analysis, patches vLLM source code, optionally gathers Nsight Compute kernel evidence, and revalidates on the real model.
  • Model PR history gate: Consults model-family optimization history before choosing source paths, recording PR state, immutable source heads, and validation evidence.
  • Use Case: Given a model like a new MoE LLM on H100 GPUs, the Skill benchmarks all frameworks, finds vLLM 8% behind SGLang in decode, profiles both, patches the vLLM attention kernel path, and iterates until vLLM ties or wins under the fixed SLA.

Quick Start

Ask the agent to run the vLLM SOTA Humanize loop for your model id, target GPU environment, and precision policy so it benchmarks against SGLang and TensorRT-LLM and patches vLLM until it reaches SOTA.

Frequently Asked Questions about vllm-sota-humanize-loop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I make vLLM match SGLang or TensorRT-LLM performance for a model?▼

Run this Skill with the model id and target GPU environment. It first executes a fixed fair benchmark across vLLM, SGLang, and TensorRT-LLM, then starts a Humanize RLCR loop that profiles the gap, patches vLLM code, and revalidates until vLLM ties or wins.

How does the vLLM SOTA loop decide when to stop patching?▼

The loop stops when vLLM beats the best SLA-passing competitor, ties within a stable 1% threshold after repeat runs, the gap is proven external to vLLM, or profiling shows the hot path is near hardware limits with no low-risk patch remaining.

Does this workflow require the Humanize runtime and other skills?▼

Yes. It requires the Humanize runtime providing scripts/setup-rlcr-loop.sh, plus companion skills for serving benchmarks, torch-profiler analysis, layer-pipeline analysis, and model PR history. Nsight Compute evidence via ncu-report-skill is optional for kernel patches.

Can the loop change the benchmark workload to make vLLM look faster?▼

No. The fixed workload, SLA, and competitor commands are locked after the baseline phase. Acceptance criteria explicitly fail any patch that only changes workload, request counts, SLA, or competitor commands instead of vLLM code.

When is Nsight Compute kernel evidence used in the loop?▼

NCU evidence is used only when vLLM remains over 1% behind, the slow stage has a kernel or kernel family with at least 1% GPU-time share, layer-pipeline analysis identified the target, and torch-profiler evidence is insufficient to choose the next edit.

What happens if the optimization campaign is interrupted?▼

The campaign is recoverable from the run artifact root. The model-loop checkpoint, ledgers, benchmark and profile artifacts, and NCU digests preserve enough lineage to resume without losing benchmark, profile, or patch history.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
3.7k

deployment

Serve quantized or unquantized LLM checkpoints as OpenAI-compatible endpoints using vLLM, SGLang, or TRT-LLM.

Official
Advanced
QingHuan-6QingHuan-6

sglang-sota-performance

Compare SGLang, vLLM, and TensorRT-LLM deployment commands and generate benchmark reports.

Community
Advanced
anderssonqanderssonq
1

nvsinner-failure-archaeology

Preserve NvSinner failure investigation history and rejected approaches.

Community
Advanced
zulumonkeymetalliczulumonkeymetallic

evaluating-llms-harness

Benchmark and compare LLMs across standardized evaluation tasks.

Community
Advanced