skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

benchmark-model-kernels

Derives per-rank GEMM and fused-MoE shapes from Hugging Face models and runs FlashInfer microbenchmarks.

3.7k|576|Updated Apr 23, 2024
Search Tags:#benchmark#FlashInfer#moe#llm inference#nvfp4#GEMM#Tensor Parallelism
One-click install
npx skills add https://github.com/NVIDIA/Model-Optimizer --skill benchmark-model-kernels
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: benchmark-model-kernels
Source: https://github.com/NVIDIA/Model-Optimizer/tree/main/plugins/modelopt/skills/benchmark-model-kernels
Command: npx skills add https://github.com/NVIDIA/Model-Optimizer --skill benchmark-model-kernels

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, accelerate, flashinfer, numpy, pynvml, vllm, and includes scripts (resource) components.

What problem does it solve?

Choosing a model, GPU, or TP/EP layout for LLM serving requires knowing how the per-rank GEMM and fused-MoE kernels perform under BF16, FP8, and NVFP4, but deriving those shapes normally means loading full checkpoints and hand-computing sharding. This Skill inspects a Hugging Face model on meta tensors (no weights loaded), derives the fused QKV, gate/up, Mamba, GatedDeltaNet, and routed-expert shapes for a given TP/EP, and runs FlashInfer kernel microbenchmarks for a chosen M sweep.

Core Features & Use Cases

  • Shape derivation without checkpoints: Builds the model on meta tensors from config alone, applies vLLM-style TP/EP sharding rules, and prints a preview of every per-rank GEMM and MoE shape before any GPU work.
  • FlashInfer kernel benchmarking: Runs BF16, FP8, and NVFP4 GEMM and fused-MoE cases through a local FlashInfer source checkout, including activation-quantization timing and vLLM-equivalent physical padding.
  • Audited coverage: Detects unsupported decoder Linear layouts and reports them explicitly, with a manual-supplement path via benchmark_via_builtin.py for missing shapes.
  • Use Case: Before deploying a 550B MoE model with NVFP4 on 8 GPUs, preview the per-rank shapes with TP=8/EP=8, then benchmark M values 1, 8, 64, 512 to compare CUTLASS, trtllm-gen, and CuteDSL MoE backends.

Quick Start

Ask the assistant to benchmark the per-rank kernels of a Hugging Face model such as 'meta-llama/Llama-3.1-8B' with TP=1, EP=1, and the default M sweep, starting with the shape preview.

Frequently Asked Questions about benchmark-model-kernels

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark LLM GEMM kernels without loading model weights?▼

Build the Hugging Face model on meta tensors from its config alone, which instantiates all Linear shapes without allocating memory. The benchmark_model.py script then derives fused QKV, gate/up, and expert shapes and runs FlashInfer microbenchmarks for those shapes.

How do I derive per-rank shapes for tensor and expert parallelism?▼

Pass --tp and --ep to benchmark_model.py and it applies vLLM's sharding rules: TP shards attention heads and projection widths, while EP partitions routed experts across ranks. The script validates divisibility, GQA replication, and top_k against the per-rank expert count and errors loudly on invalid layouts.

Does FlashInfer benchmarking require the installed package or a source checkout?▼

A FlashInfer source checkout containing benchmarks/flashinfer_benchmark.py is required; the installed wheel alone is not enough. Prefer a clean checkout matching the installed flashinfer version, and verify CUPTI timing works before running full benchmarks.

Why does the NVFP4 mm_fp4 benchmark fail with a misaligned address error?▼

The trtllm mm_fp4 backend requires N divisible by 128 for its shuffled weight layout, and gated NVFP4 CUTLASS MoE fails when 2F is not divisible by 128 per rank. Prefer EP over TP for experts to keep per-rank widths legal rather than benchmarking padded shapes.

Can these kernel benchmarks measure end-to-end serving latency or throughput?▼

No, these are kernel-only timings that omit weights, layer frequency, communication, KV cache, and scheduling. They should never be presented as end-to-end latency or throughput; use a serving framework's own benchmarks for those metrics.

Related Skills

Explore More Software Engineering Skills→
jwu1980-testjwu1980-test

benchmark-kernel

Benchmark FlashInfer GPU kernels with CUPTI timing and output median throughput metrics.

Official
Intermediate
eddieluengeddielueng
1

tensorrt-llm

Compile and run LLM inference with NVIDIA TensorRT-LLM.

Community
Advanced
RediSearchRediSearch
6.2k

run-macro-benchmarks

Run RediSearch end-to-end macro benchmarks against a real redis-server via redisbench-admin.

Official
Intermediate
NVIDIANVIDIA
14.5k

perf-analyze

Diagnoses TensorRT-LLM serving deployments via benchmarking, SOL projection, and kernel-level profiling.

Official
Advanced