skills.rest
Z43L
Z43LCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Deploys TensorRT-LLM to optimize LLM inference on NVIDIA GPUs.

Updated May 5, 2026
Search Tags:#quantization#low latency#nvidia#model serving#llm inference#tensorrt-llm#multi-gpu
One-click install
npx skills add https://github.com/Z43L/zeus-agent --skill tensorrt-llm-z43l
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Z43L/zeus-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/Z43L/zeus-agent --skill tensorrt-llm-z43l

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM reduces the cost of running large language models by delivering significantly lower latency and higher throughput compared to standard PyTorch inference on NVIDIA hardware.

Core Features & Use Cases

  • Optimized LLM inference on NVIDIA GPUs: Uses TensorRT kernels for performance-focused execution.
  • Quantization for production efficiency: Supports FP8 and INT4/FP4 to reduce memory footprint while improving speed.
  • High-throughput serving: Enables in-flight batching, KV cache optimizations, and OpenAI-compatible API serving via trtllm-serve.
  • Scaling across GPUs: Supports tensor/pipeline/expert parallelism and multi-node deployment for large models.

Quick Start

Use the tensorrt-llm skill to start an OpenAI-compatible chat server for a selected model by running: trtllm-serve meta-llama/Meta-Llama-3-8B --tp_size 1 --max_batch_size 256 --port 8000

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference on NVIDIA GPUs for lower latency?▼

Yes, TensorRT-LLM supports production efficiency through FP8 and INT4 quantization, reducing memory footprint while maintaining performance for large language models.

How do I start an OpenAI-compatible chat server using TensorRT-LLM?▼

High-throughput LLM serving in TensorRT-LLM uses dynamic in-flight batching and KV cache optimizations to process concurrent client requests efficiently, maximizing overall GPU utilization.

Can I scale LLM serving across multiple GPUs or nodes with TensorRT-LLM?▼

TensorRT-LLM supports multi-GPU and multi-node scaling for large models using tensor, pipeline, and expert parallelism to distribute workloads across available NVIDIA hardware resources.

What are the runtime requirements for deploying models with TensorRT-LLM?▼

Runtime requirements for TensorRT-LLM deployments include specific versions of CUDA and TensorRT, alongside Python dependencies like PyTorch, to ensure proper kernel execution and build compatibility.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
13.2k

trt-cpp-runtime-quickstart

Load and run TensorRT engines with the modern C++ Runtime API.

Official
Intermediate
hsliuustc0106hsliuustc0106
84

vllm-omni-api

Bridge client applications with the vLLM-Omni API using OpenAI-compatible endpoints.

Community
Intermediate
thisismynewfmail-uithisismynewfmail-ui

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT-LLM on A100/H100 GPUs.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced