tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT compilation and quantization.

14|5|Updated Apr 9, 2026

Search Tags:#quantization #fp8 #model serving #llm inference #tensorrt-llm #nvidia gpu #multi-gpu

Ready to run

npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill tensorrt-llm-mlt-oss

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill tensorrt-llm-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Serving large language models in production with PyTorch is slow and expensive, often failing to meet throughput and latency targets on NVIDIA GPU infrastructure. ## Core Features & Use Cases - High-Performance Inference: Compile and serve models with in-flight batching, paged KV cache, and CUDA graphs for up to 100x faster inference than PyTorch. - Quantization Support: Deploy FP8, INT4 AWQ, and INT4 GPTQ quantized models to cut memory usage by 50-75% with minimal accuracy loss. - Multi-GPU Scaling: Scale models like Llama 3-405B across GPUs and nodes using tensor, pipeline, and expert parallelism. - Use Case: Deploy Llama 3-70B on 4 A100 GPUs with FP8 quantization and serve it through an OpenAI-compatible API with Prometheus metrics and Kubernetes autoscaling. ## Quick Start Use the tensorrt-llm skill to serve Llama 3-8B with trtllm-serve on a single GPU with FP8 quantization enabled.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I serve a Llama model with TensorRT-LLM?▼

Use the trtllm-serve command with the HuggingFace model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B with flags for tp_size, max_batch_size, and dtype. It starts an OpenAI-compatible server on port 8000 that handles model download and compilation automatically.

TensorRT-LLM vs vLLM for LLM inference?▼

TensorRT-LLM delivers maximum throughput and lowest latency on NVIDIA GPUs through compilation, FP8 quantization, and CUDA graphs. vLLM offers a simpler Python-first setup with PagedAttention and works on non-NVIDIA hardware, making it better for quick prototyping.

Does TensorRT-LLM support FP8 and INT4 quantization?▼

Yes, TensorRT-LLM supports FP8, INT4 AWQ, and INT4 GPTQ quantization. FP8 on H100 gives roughly 2x speedup with 50% memory reduction, while INT4 provides 3-4x speedup with 75% memory savings at a small accuracy cost.

How do I run a 70B model across multiple GPUs?▼

Set tensor_parallel_size to split the model across GPUs, for example tensor_parallel_size=4 for Llama 3-70B on four A100 80GB GPUs with FP8. NVLink is strongly recommended for tensor parallelism, and pipeline parallelism handles multi-node deployments.

Why does TensorRT-LLM run out of GPU memory?▼

OOM occurs when batch size, max_num_tokens, or model size exceeds available VRAM. Reduce max_batch_size, enable FP8 or INT4 quantization, increase tensor_parallel_size, or enable chunked context for long prompts.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPU, Apple Silicon, or AMD GPUs, where llama.cpp or vLLM are better fits. It also adds compilation complexity, so simpler tools may suit lightweight experimentation.