skills.rest
Jensen-Yao
Jensen-YaoCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT with quantization and multi-GPU parallelism.

2|Updated Aug 15, 2026
Search Tags:#quantization#LLM inference#FP8#multi-GPU#model serving#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/Jensen-Yao/agents-skills --skill tensorrt-llm-jensen-yao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Jensen-Yao/agents-skills/tree/main/skills/tensorrt-llm
Command: npx skills add https://github.com/Jensen-Yao/agents-skills --skill tensorrt-llm-jensen-yao

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Serving large language models in production with PyTorch is slow and expensive, often failing to meet throughput and latency targets on NVIDIA GPU infrastructure. ## Core Features & Use Cases - High-Performance Inference: Achieves up to 100x faster inference than PyTorch using in-flight batching, paged KV cache, Flash Attention, and CUDA graphs. - Quantization Support: Runs FP8, INT4 (AWQ/GPTQ), and FP4 quantized models for 2-4x speedups and major memory savings. - Multi-GPU & Production Serving: Scales models like Llama 3-405B across GPUs and nodes with tensor/pipeline/expert parallelism, and serves them via an OpenAI-compatible API with Prometheus metrics and Kubernetes deployment patterns. - Use Case: Deploy Llama 3-70B on 4 A100 GPUs with FP8 quantization and trtllm-serve to handle thousands of concurrent chat requests at 10,000+ tokens/sec. ## Quick Start Deploy the Llama 3-8B model as an OpenAI-compatible API server on my NVIDIA GPUs using TensorRT-LLM with FP8 quantization.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve an LLM with TensorRT-LLM?▼

Use the trtllm-serve command with your HuggingFace model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B --tp_size 4 --max_batch_size 256. It starts an OpenAI-compatible API server on port 8000 that handles model download and compilation automatically.

TensorRT-LLM vs vLLM for LLM inference?▼

TensorRT-LLM delivers maximum throughput and lowest latency on NVIDIA GPUs through TensorRT compilation and FP8/INT4 quantization. vLLM offers a simpler Python-first setup with PagedAttention and works on non-NVIDIA hardware like AMD GPUs.

Does TensorRT-LLM support FP8 quantization?▼

Yes, TensorRT-LLM supports FP8 quantization by setting dtype="fp8" when initializing the LLM, delivering roughly 2x faster inference and 50% memory reduction on H100 GPUs with under 1% perplexity degradation. INT4 AWQ and GPTQ are also supported.

How do I run a 70B model across multiple GPUs?▼

Set tensor_parallel_size when creating the LLM instance, for example tensor_parallel_size=4 to shard Llama 3-70B across 4 A100 GPUs with FP8. For 405B models, combine tensor parallelism within nodes and pipeline parallelism across nodes.

Why does TensorRT-LLM run out of GPU memory?▼

OOM errors occur when batch size or token limits exceed available memory. Reduce max_batch_size and max_num_tokens, enable FP8 or INT4 quantization, or increase tensor_parallel_size to distribute the model across more GPUs.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPU or Apple Silicon, where llama.cpp with GGUF quantization is appropriate, or when you need a simpler Python-first setup without TensorRT compilation, where vLLM is a better fit.

Related Skills

Explore More Software Engineering Skills→
JuanMS20JuanMS20

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
tadod12tadod12

serving-llms-vllm

Serve LLM inference via OpenAI-compatible endpoints using vLLM.

Community
Advanced
DeepLink-orgDeepLink-org
7

nvidia-nlp-inference

Benchmarks SGLang-served HuggingFace text models on NVIDIA GPUs for throughput and latency.

Official
Advanced
moltis-orgmoltis-org
2.8k

llama-cpp

Run local LLM inference on CPU and GPU with llama.cpp and GGUF models.

Official
Advanced