skills.rest
loteiron
loteironCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Deploy and optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated Sep 10, 2026
Search Tags:#quantization#LLM inference#FP8#multi-GPU#model serving#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/loteiron/ZeusAgent --skill tensorrt-llm-loteiron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/loteiron/ZeusAgent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/loteiron/ZeusAgent --skill tensorrt-llm-loteiron

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Serving large language models in production demands high throughput, low latency, and efficient GPU memory usage, which naive PyTorch deployments cannot deliver on NVIDIA hardware. ## Core Features & Use Cases - Optimized Inference: Run models like Llama 3 with in-flight batching, paged KV cache, and CUDA graphs for up to 24,000 tokens/sec on H100 GPUs. - Quantization Support: Deploy FP8, INT4 AWQ, or GPTQ quantized models to cut memory usage by 50-75% with minimal accuracy loss. - Multi-GPU Scaling: Use tensor, pipeline, and expert parallelism to serve 70B-405B parameter models across multiple GPUs or nodes. - Use Case: Serve Llama 3-70B behind an OpenAI-compatible API on 4 A100 GPUs with FP8 quantization, Prometheus metrics, and Kubernetes autoscaling. ## Quick Start Use the tensorrt-llm skill to serve meta-llama/Meta-Llama-3-8B with trtllm-serve on 4 GPUs using FP8 quantization.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve Llama 3 with TensorRT-LLM?▼

Use trtllm-serve with the model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B with flags like --tp_size and --max_batch_size. It starts an OpenAI-compatible server on port 8000 that handles model download and compilation automatically.

TensorRT-LLM vs vLLM for LLM inference?▼

TensorRT-LLM delivers maximum throughput on NVIDIA GPUs through TensorRT compilation and FP8/INT4 quantization, while vLLM offers a simpler Python-first setup with PagedAttention. Choose vLLM for AMD GPUs or non-NVIDIA hardware.

Does TensorRT-LLM support FP8 quantization?▼

Yes, TensorRT-LLM supports FP8 quantization by passing dtype="fp8" when initializing the LLM. On H100 GPUs this provides roughly 2x faster inference and 50% memory reduction with under 1% perplexity degradation.

Can TensorRT-LLM run on multiple GPUs?▼

Yes, it supports tensor parallelism, pipeline parallelism, and expert parallelism for MoE models. Set tensor_parallel_size in the LLM API or --tp_size in trtllm-serve, and combine with pipeline parallelism for multi-node deployments of 405B-class models.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPU, Apple Silicon, or AMD GPUs, since it requires NVIDIA hardware. Use llama.cpp for CPU or edge deployment with GGUF models, and vLLM when you want simpler setup without TensorRT compilation.

Why does TensorRT-LLM run out of GPU memory?▼

OOM occurs when batch size, token limits, or model size exceed available VRAM. Reduce max_batch_size and max_num_tokens, enable FP8 or INT4 quantization, or increase tensor_parallel_size to shard the model across more GPUs.

Related Skills

Explore More Software Engineering Skills→
matlee0409matlee0409

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
tadod12tadod12

serving-llms-vllm

Serve LLM inference via OpenAI-compatible endpoints using vLLM.

Community
Advanced
agentic-inagentic-in
577

Inference

Select inference paths, serving stacks, and benchmarking plans for ML models.

Official
Intermediate
moltis-orgmoltis-org
2.8k

llama-cpp

Run local LLM inference on CPU and GPU with llama.cpp and GGUF models.

Official
Advanced