skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

trtllm-test-specialist

Classifies and runs TensorRT-LLM module, benchmark, and evaluation tests via delegated workflows.

14.5k|2.7k|Updated Aug 16, 2023
Search Tags:#evaluation#benchmark#GPU#Testing#PyTest#tensorRT-llm#Slurm
One-click install
npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill trtllm-test-specialist
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trtllm-test-specialist
Source: https://github.com/NVIDIA/TensorRT-LLM/tree/main/.claude/skills/trtllm-test-specialist
Command: npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill trtllm-test-specialist

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pyyaml, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Running TensorRT-LLM tests requires choosing the right workflow (module pytest, trtllm-bench, trtllm-eval, or perf-sanity), building correct commands with parallelism flags, and handling single-node versus multi-node constraints. This Skill automates that classification, command construction, and execution delegation.

Core Features & Use Cases

  • Test Scope Classification: Automatically routes requests to module tests, functionality smoke tests, benchmark tests, or accuracy evaluation tests based on inputs like test_cmd, eval_tasks, or mtp_layers.
  • Command Generation: Builds trtllm-bench, trtllm-eval, and perf-sanity pytest commands with correct tp/pp/ep flags, dataset preparation, and option placement rules.
  • Multi-Node Fallback: Generates perf-sanity config YAMLs (aggregated or disaggregated) when workloads exceed one node, including MTP speculative decoding sweeps.
  • Use Case: Ask to benchmark Llama-3.1-8B with tp_size 4 on a B200 node; the Skill resolves parameters, builds the trtllm-bench throughput command, delegates execution, and writes a markdown report.

Quick Start

Run a functionality smoke test for the Llama-3.1-8B-Instruct checkpoint at /models/Llama-3.1-8B-Instruct with tensor parallelism 4.

Frequently Asked Questions about trtllm-test-specialist

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a TensorRT-LLM benchmark test?▼

Provide the model name, checkpoint path, and parallelism sizes; the Skill builds a trtllm-bench throughput or latency command with a synthetic dataset and delegates execution. Single-node runs use trtllm-bench directly, while multi-node runs fall back to a generated perf-sanity config YAML.

How do I run accuracy evaluation with trtllm-eval?▼

Specify eval_tasks such as gsm8k or mmlu along with the checkpoint path and tp_size. The Skill builds one trtllm-eval command per task with correct global and per-task option placement, then executes and reports accuracy results.

Does trtllm-bench support multi-node benchmarking?▼

No, trtllm-bench is treated as single-node only. When the world size exceeds the GPUs available on one node, the Skill generates a perf-sanity config YAML and runs the benchmark through the pytest-based perf-sanity path instead.

What happens when I provide a config file to the test workflow?▼

The YAML config file becomes the sole source of parameters and all directly specified values are dropped with a warning. The parse_config.py script validates the file, applies defaults, and reports any missing required parameters before execution.

What are the GPU limits for module-level pytest runs?▼

Module tests are limited to a maximum of 8 GPUs and a single node. The extract_test_markers.py script derives required_devices and device_type from pytest markers like skip_less_device, and execution stops if limits are exceeded.

How do I benchmark MTP speculative decoding layers?▼

Provide mtp_layers as a comma-separated list such as 1,2,3 to trigger a sweep. The Skill generates one perf-sanity config per MTP value and submits all jobs concurrently for cluster execution.

Related Skills

Explore More Software Engineering Skills→
Supporter09Supporter09

evaluating-cosmos-policy

Runs Cosmos Policy LIBERO and RoboCasa benchmark evaluations via command-line workflows.

Community
Advanced
jiunbaejiunbae
12

ml-benchmark

Automate ML model benchmarking across versions and languages with JSON and Markdown reports.

Community
Advanced
NVIDIA-TAONVIDIA-TAO
83

tao-run-on-kubernetes

Submit TAO container jobs as Kubernetes Jobs with NVIDIA GPU scheduling.

Official
Advanced
busboom-software-studiobusboom-software-studio
1

oop

Apply small targeted changes directly on master with test suite runs.

Official
Intermediate