What problem does it solve? Evaluating LLM inference performance on NVIDIA GPUs requires coordinating container setup, SGLang server lifecycle, offline load testing, and metric extraction, which is error-prone when done manually. This Skill standardizes the entire workflow so results are comparable across models and runs. ## Core Features & Use Cases - Unified serve/bench/calc pipeline: Prebuilt scripts launch SGLang servers, run fixed-seed offline benchmarks with sglang.bench_serving, and parse logs into a schema-versioned result.json with throughput, TTFT, TPOT, ITL, and end-to-end latency metrics. - Model profiles and multi-host support: A profile registry (generic, DeepSeek-R1, Llama-1/2 variants) defines per-model TP, sequence lengths, and precision matrices, with rank-aware multi-node execution over NCCL/NVSHMEM. - Use Case: Ask the agent to evaluate DeepSeek-R1 inference on an 8-GPU node; the Skill mounts model and dataset directories into the container, serves the model with TP=8, runs 1000 prompts at 2048 input/output tokens, and produces a validated result.json. ## Quick Start Ask the agent to evaluate the inference throughput and latency of your HuggingFace model on NVIDIA GPUs using the nvidia-nlp-inference skill with the mounted model and dataset directories.
