serve-config-guide

What problem does it solve?

Generates a repo-grounded starting YAML for trtllm-serve --config that enables predictable single-node PyTorch serving and aligns with checked-in TensorRT-LLM configurations and deployment docs, preserving explicit latency / balanced / throughput objectives and excluding disaggregated, multi-node, and non-MTP speculative configs.

Core Features & Use Cases

• Repo-grounded starting config for single-node PyTorch serving with in-flight batching compatibility.
• Preserves latency objectives (Min Latency, Balanced, Max Throughput) and excludes disaggregated/multi-node/speculative configurations.
• Guides users to adjust common knobs (max_batch_size, max_seq_len, kv_cache_config, etc.) using checked-in sources and deployment docs.

Quick Start

Clone the repository and load the appropriate starting trtllm-serve config for your model and GPU, then adjust knobs per the model guide.