skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

serve-config-guide

Generate repo-grounded trtllm-serve YAML configs for single-node PyTorch serving.

2.8k|332|Updated Feb 25, 2026
Search Tags:#yaml#in-flight-batching#single-node#serve-config#trtllm#pytorch-serving#config-tuning
One-click install
npx skills add https://github.com/NVIDIA/skills --skill serve-config-guide
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: serve-config-guide
Source: https://github.com/NVIDIA/skills/tree/main/skills/TensorRT-LLM/serve-config-guide
Command: npx skills add https://github.com/NVIDIA/skills --skill serve-config-guide

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Generates a repo-grounded starting YAML for trtllm-serve --config that enables predictable single-node PyTorch serving and aligns with checked-in TensorRT-LLM configurations and deployment docs, preserving explicit latency / balanced / throughput objectives and excluding disaggregated, multi-node, and non-MTP speculative configs.

Core Features & Use Cases

  • Repo-grounded starting config for single-node PyTorch serving with in-flight batching compatibility.
  • Preserves latency objectives (Min Latency, Balanced, Max Throughput) and excludes disaggregated/multi-node/speculative configurations.
  • Guides users to adjust common knobs (max_batch_size, max_seq_len, kv_cache_config, etc.) using checked-in sources and deployment docs.

Quick Start

Clone the repository and load the appropriate starting trtllm-serve config for your model and GPU, then adjust knobs per the model guide.

Frequently Asked Questions about serve-config-guide

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate a trtllm-serve YAML config for single-node PyTorch serving?▼

You generate a trtllm-serve YAML config by reading model docs and checked-in TensorRT-LLM configurations to create a deployment-ready starting file for single-node PyTorch serving with in-flight batching compatibility.

Can I use trtllm-serve configs for disaggregated or multi-node TensorRT-LLM deployments?▼

No, trtllm-serve configs generated here exclude disaggregated, multi-node, and non-MTP speculative configurations, focusing solely on single-node PyTorch serving with aggregate and in-flight batching workloads.

What latency objectives are preserved when generating a TensorRT-LLM serving config?▼

The generated serving config preserves explicit latency objectives including Min Latency, Balanced, and Max Throughput, aligning your deployment with checked-in TensorRT-LLM sources and model documentation.

How do I tune max_batch_size and kv_cache settings for in-flight batching?▼

You tune max_batch_size, max_seq_len, and kv_cache_config by adjusting source-backed fields within the generated YAML, using guidance from checked-in configs and deployment docs to match your model and GPU.

Does this serving config generation support DeepSeek-R1 MTP configurations?▼

Yes, DeepSeek-R1 MTP configurations are supported specifically when exact checked-in configs exist in the repository, ensuring the generated trtllm-serve YAML aligns with verified TensorRT-LLM deployment sources.

What are the limitations of using generated trtllm-serve configs for speculative decoding?▼

The generated trtllm-serve configs strictly exclude non-speculative decoding defaults and disaggregated setups, limiting deployment scope to single-node aggregate and in-flight batching workloads without speculative execution.

Related Skills

Explore More Software Engineering Skills→
yo-stevenyo-steven

trtllm-serve-config-guide

Generate source-backed trtllm-serve YAML configs for single-node PyTorch serving.

Community
Advanced
ZardLi1115ZardLi1115
4

tensorrt-llm

Compile and serve LLMs on NVIDIA GPUs with TensorRT-LLM.

Community
Intermediate
informatico-madridinformatico-madrid
1

tensorrt-llm

Optimizes LLM serving on NVIDIA hardware with batched GPU inference.

Community
Advanced
gagan114662gagan114662

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced