skills.rest
Tgoldi
TgoldiCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT compilation, quantization, and multi-GPU parallelism.

1|Updated May 10, 2026
Search Tags:#quantization#fp8#model serving#llm inference#tensorrt-llm#nvidia gpu#multi-gpu
One-click install
npx skills add https://github.com/Tgoldi/claude-skills --skill tensorrt-llm-tgoldi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Tgoldi/claude-skills/tree/main/tensorrt-llm
Command: npx skills add https://github.com/Tgoldi/claude-skills --skill tensorrt-llm-tgoldi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Deploying large language models in production with PyTorch alone delivers poor throughput and high latency, driving up GPU costs and degrading user experience. This Skill guides you through compiling, quantizing, and serving LLMs with NVIDIA TensorRT-LLM to achieve 10-100x faster inference on A100 and H100 hardware. ## Core Features & Use Cases - Quantization & Optimization: Apply FP8, INT4 (AWQ/GPTQ) quantization, in-flight batching, paged KV cache, speculative decoding, and CUDA graphs to maximize throughput and cut memory usage by 50-75%. - Multi-GPU & Multi-Node Scaling: Configure tensor, pipeline, and expert parallelism to serve models from 8B to 405B parameters across NVLink-connected GPUs and InfiniBand clusters. - Production Serving: Deploy an OpenAI-compatible API with trtllm-serve, complete with Prometheus metrics, health probes, Kubernetes manifests, autoscaling, and load balancing. - Use Case: You need to serve Llama 3-70B to thousands of concurrent users. Use this Skill to quantize the model to FP8, shard it across 4 A100 GPUs with tensor parallelism, and expose it behind an autoscaling Kubernetes deployment with monitoring. ## Quick Start Use the tensorrt-llm skill to serve Llama 3-8B with FP8 quantization on my H100 GPU with an OpenAI-compatible endpoint.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve Llama 3 with TensorRT-LLM?▼

Run trtllm-serve with the HuggingFace model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B --tp_size 4 --max_batch_size 256. The server automatically downloads and compiles the model, then exposes an OpenAI-compatible API on port 8000.

TensorRT-LLM vs vLLM: which should I use for inference?▼

TensorRT-LLM delivers maximum throughput on NVIDIA GPUs through compilation and FP8/INT4 quantization, making it ideal for production on A100/H100. vLLM offers a simpler Python-first setup with PagedAttention and works on non-NVIDIA hardware.

Does TensorRT-LLM support FP8 quantization on H100?▼

Yes, FP8 quantization is supported and recommended on H100 GPUs. Set dtype="fp8" when initializing the LLM class to get roughly 2x faster inference and 50% memory reduction with under 1% perplexity degradation.

How do I run a 70B model across multiple GPUs with TensorRT-LLM?▼

Use tensor parallelism by setting tensor_parallel_size=4 in the LLM constructor or --tp_size 4 in trtllm-serve. This shards Llama 3-70B across 4 A100 80GB GPUs with FP8, using about 35GB per GPU, and NVLink is recommended for best performance.

Why am I getting OOM errors with TensorRT-LLM?▼

OOM errors occur when batch size, token limits, or model size exceed GPU memory. Reduce max_batch_size and max_num_tokens, enable FP8 or INT4 quantization, or increase tensor_parallel_size to distribute the model across more GPUs.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPU, Apple Silicon, or AMD GPUs, since it requires NVIDIA hardware and CUDA. For edge or CPU deployment use llama.cpp with GGUF models, and for simpler Python-first serving consider vLLM.

Related Skills

Explore More Software Engineering Skills→
wshobsonwshobson
39.3k

finetuning-method-selection

Routes fine-tuning requests to the right method, base model size class, and memory plan.

Community
Intermediate
understudylabsunderstudylabs
10

run-local-model-lab

Compare local MLX models against frontier gateways on latency, cost, and quality.

Official
Advanced
cryptopaficryptopafi

serving-llms-vllm

Serve large language models with high-throughput inference for OpenAI-compatible endpoints.

Community
Advanced
moltis-orgmoltis-org
2.8k

llama-cpp

Run local LLM inference on CPU and GPU with llama.cpp and GGUF models.

Official
Advanced