skills.rest
qcmuu
qcmuuCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize and serve LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated May 2, 2026
Search Tags:#quantization#llm inference#multi-gpu#tensorRT#trtllm-serve#openai compatible api#paged kv cache
One-click install
npx skills add https://github.com/qcmuu/AI-Research-Skills --skill tensorrt-llm-qcmuu
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/qcmuu/AI-Research-Skills/tree/main/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/qcmuu/AI-Research-Skills --skill tensorrt-llm-qcmuu

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM eliminates slow, memory-heavy LLM inference by compiling optimized execution paths for NVIDIA GPUs, so your service delivers low latency and high throughput.

Core Features & Use Cases

  • In-flight batching for higher utilization under mixed request lengths (faster tokens/sec without waiting for all sequences to finish).
  • Paged KV cache to improve long-context memory efficiency and throughput stability.
  • Quantization support (FP8/INT4/FP4) to run larger models with lower GPU memory and faster execution, with multi-GPU scaling via tensor/pipeline/expert parallelism.
  • Use case: Deploy an Llama 3 model on A100/H100 with FP8 or INT4 and serve OpenAI-compatible /v1/chat/completions for real-time production workloads.

Quick Start

Start a TensorRT-LLM server for your model by running: "trtllm-serve meta-llama/Meta-Llama-3-8B --tp_size 1 --max_batch_size 256 --max_num_tokens 4096".

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve LLM inference with low latency and high throughput on NVIDIA GPUs?▼

To serve LLM inference with low latency, use TensorRT-LLM to compile optimized execution paths for NVIDIA GPUs. It provides in-flight batching and paged KV cache to maintain high throughput during real-time chat and long-context generation.

What's the best way to run larger LLMs with limited GPU memory?▼

The best way to run larger LLMs with limited GPU memory is applying FP8 or INT4 quantization. TensorRT-LLM supports these formats to reduce memory usage and accelerate execution while scaling across multiple GPUs using tensor parallelism.

Can I deploy an OpenAI-compatible API for production Llama 3 workloads?▼

Yes, you can deploy an OpenAI-compatible API for production Llama 3 workloads using the trtllm-serve command. It exposes an HTTP /v1/chat/completions endpoint optimized for real-time production deployment on A100 or H100 GPUs.

Does paged KV cache improve long-context generation throughput?▼

Paged KV cache improves long-context generation throughput by optimizing memory efficiency. TensorRT-LLM uses this mechanism to stabilize throughput levels when handling extended context lengths during inference.

How do I start serving a model with multi-GPU tensor parallelism?▼

Start serving a model with multi-GPU tensor parallelism by running trtllm-serve with the --tp_size flag. This configures the compiled execution engine to distribute the model across multiple NVIDIA GPUs for faster inference.

Related Skills

Explore More Software Engineering Skills→
DarkArty07DarkArty07
3

llama-cpp

Deploy local LLM inference with GGUF quantized weights via llama.cpp.

Community
Advanced
NaimalArain13NaimalArain13
2

chatkit-python

Build a FastAPI chat backend with SSE streaming and conversation persistence.

Community
Intermediate
zhuangbiaoweizhuangbiaowei
2

training-llms-megatron

Train large-scale language models using NVIDIA Megatron-Core with advanced parallelism.

Community
Advanced
helix4uhelix4u

llama-cpp

Run LLM inference on CPU, Apple Silicon, and non-NVIDIA GPUs with GGUF models.

Community
Intermediate