skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT compilation and quantization.

Updated Jun 5, 2026
Search Tags:#quantization#fp8#model serving#tensor parallelism#llm inference#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/xu1713/openhorse --skill tensorrt-llm-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/xu1713/openhorse --skill tensorrt-llm-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve? Deploying large language models in production often suffers from high latency, low throughput, and excessive GPU memory consumption when using standard PyTorch inference. This Skill configures NVIDIA TensorRT-LLM to compile and serve models with dramatically faster inference, lower memory footprint, and efficient multi-GPU scaling. ## Core Features & Use Cases - Quantized Inference: Run models in FP8, INT4 (AWQ/GPTQ), or FP4 for 2-4x faster generation and up to 75% memory reduction. - Production Serving: Launch an OpenAI-compatible API server with in-flight batching, paged KV cache, streaming, Prometheus metrics, and Kubernetes deployment manifests. - Multi-GPU Scaling: Configure tensor, pipeline, and expert parallelism to serve models from 8B to 405B parameters across multiple GPUs or nodes. - Use Case: Serve Llama 3-70B on 4 A100 GPUs with FP8 quantization and tensor parallelism, achieving over 10,000 tokens per second behind an OpenAI-compatible endpoint with autoscaling. ## Quick Start Use the tensorrt-llm skill to serve the Llama 3 8B model with FP8 quantization on my available GPUs and expose an OpenAI-compatible chat completions endpoint.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I serve Llama 3 with TensorRT-LLM on multiple GPUs?▼

Use trtllm-serve with the --tp_size flag to enable tensor parallelism across GPUs, for example trtllm-serve meta-llama/Meta-Llama-3-70B --tp_size 4 --dtype fp8. This shards the model across 4 GPUs and exposes an OpenAI-compatible endpoint on port 8000.

TensorRT-LLM vs vLLM for LLM inference serving?▼

TensorRT-LLM delivers maximum throughput and lowest latency on NVIDIA GPUs through compilation, FP8/INT4 quantization, and CUDA graphs. vLLM offers a simpler Python-first setup with PagedAttention and works on non-NVIDIA hardware, making it better for rapid iteration.

Does TensorRT-LLM support FP8 and INT4 quantization?▼

Yes, TensorRT-LLM supports FP8, INT4 AWQ, INT4 GPTQ, and FP4 quantization. FP8 on H100 provides roughly 2x speedup with 50% memory reduction and minimal accuracy loss, while INT4 gives 3-4x speedup when memory is critical.

Can I run TensorRT-LLM on AMD GPUs or CPU?▼

No, TensorRT-LLM requires NVIDIA GPUs such as A100, H100, or GB200 with CUDA. For AMD GPUs or CPU deployment, use alternatives like vLLM for broader hardware support or llama.cpp for CPU and Apple Silicon inference.

Why does TensorRT-LLM run out of memory during serving?▼

OOM occurs when batch size, token limits, or model size exceed available GPU memory. Reduce max_batch_size and max_num_tokens, enable FP8 or INT4 quantization, or increase tensor_parallel_size to shard the model across more GPUs.

How do I monitor TensorRT-LLM in production?▼

Start the server with --enable_metrics --metrics_port 9090 to expose Prometheus metrics including request latency, tokens generated, queue size, and GPU memory usage. Health endpoints at /health/ready and /health/live integrate with Kubernetes probes.

Related Skills

Explore More Software Engineering Skills→
junior1pjunior1p
11

boltz

Predict biomolecular structures with Boltz-1 and Boltz-2 on local GPUs.

Community
Intermediate
AveerAveer
1

zeus-llm

Orchestrate deployment and configuration of local LLM inference servers for OpenCode.

Community
Intermediate
TheLineExpTheLineExp

perf-test

Benchmark HTTP performance and load testing across development, staging, and production environments.

Community
Advanced
moltis-orgmoltis-org
2.8k

llama-cpp

Run local LLM inference on CPU and GPU with llama.cpp and GGUF models.

Official
Advanced