skills.rest
yo-steven
yo-stevenCOMMUNITY
Explore all skills by this author →
GitHub

trtllm-serve-config-guide

Generate source-backed trtllm-serve YAML configs for single-node PyTorch serving.

Updated May 23, 2026
Search Tags:#YAML#model deployment#config generation#KV cache#trtllm-serve#latency tuning#throughput optimization
One-click install
npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill trtllm-serve-config-guide
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trtllm-serve-config-guide
Source: https://github.com/yo-steven/skills-exploration-20260522/tree/main/skills/TensorRT-LLM/trtllm-serve-config-guide
Command: npx skills add https://github.com/yo-steven/skills-exploration-20260522 --skill trtllm-serve-config-guide

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the trial-and-error of finding a good starting configuration for trtllm-serve by grounding the output YAML in the repository’s checked-in TensorRT-LLM configs and deployment guidance.

Core Features & Use Cases

  • Source-backed config generation: Produces a starting trtllm-serve --config YAML by matching the user’s model + serving constraints to checked-in configs, avoiding speculative and out-of-scope modes by default.
  • Objective-preserving selection: Keeps the user’s latency/throughput intent (e.g., Min Latency, Balanced, Max Throughput) by selecting configs using database profile labels when available.
  • Guardrailed adjustments: Updates only scenario-dependent fields (like batch/token/seq limits and KV-cache settings) after reading the relevant model deployment docs, and flags any inferred/interpolated fields as unverified.

Quick Start

Ask for a non-speculative, single-node PyTorch aggregate in-flight-batching serve YAML for your model and GPU with your target concurrency and input/output lengths, specifying whether you want Min Latency, Balanced, or Max Throughput.

Frequently Asked Questions about trtllm-serve-config-guide

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I generate a trtllm-serve YAML config for single-node PyTorch deployment?▼

You can generate a trtllm-serve YAML config by providing your model, GPU, target concurrency, input/output lengths, and a latency/throughput objective. The config is grounded in repository-checked profiles for single-node PyTorch serving.

What is the best way to tune latency and throughput for TensorRT-LLM serving?▼

Tuning latency and throughput involves selecting an objective like Min Latency, Balanced, or Max Throughput. The tool matches this intent to checked-in database profile labels and adjusts scenario-dependent fields like KV-cache settings accordingly.

Does trtllm-serve config generation support speculative decoding modes?▼

Speculative decoding modes are strictly excluded by default. The tool focuses on non-speculative, single-node aggregate in-flight-batching configs, with DeepSeek-R1 MTP serving as the standard checked-in mode.

Can I use legacy engine-build tuning values for my trtllm-serve YAML config?▼

You cannot use legacy engine-build tuning values. The tool generates a source-backed starting YAML by applying doc-informed adjustments to scenario-dependent fields like batch, token, sequence limits, and KV-cache settings.

What are the limitations when generating a TRT-LLM serve configuration?▼

Limitations include applicability only to single-node, non-disaggregated deployments and strict speculative exclusion rules. It also requires exact or near matches from the configs database to generate a valid YAML.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
2.8k

serve-config-guide

Generate repo-grounded trtllm-serve YAML configs for single-node PyTorch serving.

Official
Advanced
peteromalletpeteromallet
97

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT-LLM for throughput and latency.

Community
Advanced
skinred78skinred78
1

firebase-cloudrun

Deploy Flask apps to Google Cloud Run with Firebase Hosting URLs.

Community
Advanced