skills.rest
Chensihakniroth
ChensihaknirothCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT compilation and quantization.

Updated Jun 7, 2026
Search Tags:#quantization#LLM inference#FP8#multi-GPU#model serving#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill tensorrt-llm-chensihakniroth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Chensihakniroth/ANAKOT-AGENT/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill tensorrt-llm-chensihakniroth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Serving large language models in production with PyTorch is slow and expensive, often failing to meet throughput and latency targets on NVIDIA GPU infrastructure. ## Core Features & Use Cases - High-Throughput Inference: Achieves up to 24,000 tokens/sec on Llama 3-8B with in-flight batching, paged KV cache, and CUDA graphs. - Quantization Support: Runs FP8, INT4 AWQ, and INT4 GPTQ quantized models for 2-4x faster inference and up to 75% memory reduction. - Multi-GPU Scaling: Supports tensor, pipeline, and expert parallelism across GPUs and nodes for models up to 405B parameters. - Production Serving: Ships an OpenAI-compatible API server with Prometheus metrics, health checks, and Kubernetes deployment patterns. - Use Case: Deploy Llama 3-70B on 4 A100 GPUs with FP8 quantization and serve chat completions to thousands of concurrent users through an OpenAI-compatible endpoint. ## Quick Start Use the tensorrt-llm skill to serve the Llama 3-8B model with FP8 quantization on my NVIDIA GPU and expose an OpenAI-compatible API endpoint.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve Llama 3 with TensorRT-LLM?▼

Run trtllm-serve with the model name, for example trtllm-serve meta-llama/Meta-Llama-3-8B with flags for tp_size, max_batch_size, and dtype. It starts an OpenAI-compatible server on port 8000 that handles model download and compilation automatically.

TensorRT-LLM vs vLLM for LLM inference?▼

TensorRT-LLM delivers maximum throughput on NVIDIA GPUs through compilation and FP8 quantization but requires CUDA setup. vLLM offers a simpler Python-first API with PagedAttention and works on non-NVIDIA hardware, making it better for quick prototyping.

Does TensorRT-LLM support FP8 quantization?▼

Yes, TensorRT-LLM supports FP8 quantization by setting dtype to fp8 when initializing the LLM. On H100 GPUs this provides roughly 2x faster inference and 50% memory reduction with under 1% perplexity degradation.

Can TensorRT-LLM run on multiple GPUs?▼

Yes, it supports tensor parallelism, pipeline parallelism, and expert parallelism across multiple GPUs and nodes. Set tensor_parallel_size in the Python API or use --tp_size with trtllm-serve, with NVLink recommended for best performance.

Why does TensorRT-LLM run out of memory?▼

OOM occurs when batch size, token limits, or model size exceed GPU memory. Reduce max_batch_size and max_num_tokens, enable FP8 or INT4 quantization, or increase tensor_parallel_size to shard the model across more GPUs.

When should I not use TensorRT-LLM?▼

Avoid TensorRT-LLM when deploying on CPU, Apple Silicon, or AMD GPUs, where llama.cpp or other runtimes fit better. It is also unnecessary for simple prototypes where vLLM's easier setup outweighs raw performance gains.

Related Skills

Explore More Software Engineering Skills→
Jensen-YaoJensen-Yao
2

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT with quantization and multi-GPU parallelism.

Community
Advanced
masermediagroup-stackmasermediagroup-stack
1

model-serving

Deploy LLM and ML models for production inference with vLLM, BentoML, and streaming APIs.

Community
Advanced
DeepLink-orgDeepLink-org
7

nvidia-nlp-inference

Benchmarks SGLang-served HuggingFace text models on NVIDIA GPUs for throughput and latency.

Official
Advanced
HermesagentsHermesagents

gguf-quantization

Quantize and deploy GGUF models for llama.cpp CPU/GPU inference.

Community
Advanced