skills.rest
eddielueng
eddieluengCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Compile and run LLM inference with NVIDIA TensorRT-LLM.

1|Updated Apr 14, 2026
Search Tags:#inference optimization#llm serving#tensorRT-LLM#in-flight batching#multi-gpu parallelism#FP8 quantization
One-click install
npx skills add https://github.com/eddielueng/hermes-agent-zh --skill tensorrt-llm-eddielueng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/eddielueng/hermes-agent-zh/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/eddielueng/hermes-agent-zh --skill tensorrt-llm-eddielueng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) and assets (resource) components.

What problem does it solve?

TensorRT-LLM solves the challenge of slow and costly LLM inference by optimizing model execution on NVIDIA GPUs to achieve much higher throughput and lower latency than typical PyTorch serving.

Core Features & Use Cases

  • High-performance inference optimization: Uses in-flight batching, paged KV cache, Flash Attention kernels, CUDA graphs, and quantization (FP8/INT4/FP4) to maximize tokens/sec and reduce tail latency.
  • Scalable parallel serving: Supports tensor parallelism (TP), pipeline parallelism (PP), and expert parallelism (EP) for single-node and multi-node scaling of large models.
  • Production-ready serving: Provides an OpenAI-compatible server mode (trtllm-serve) with monitoring endpoints and operational settings for production deployments.
  • Use Case: You need real-time chat responses for a production app on NVIDIA A100/H100 hardware, and you must reduce per-request latency while increasing sustained throughput under concurrent traffic.

Quick Start

Use the tensorrt-llm skill to start an OpenAI-compatible server for a Llama 3 model by running the trtllm-serve command against meta-llama/Meta-Llama-3-8B with tensor parallel size and batching settings, then send a POST request to the /v1/chat/completions endpoint.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference to reduce latency and increase throughput on NVIDIA GPUs?▼

Optimize LLM inference by compiling models with TensorRT-LLM, which applies in-flight batching, paged KV cache, and Flash Attention kernels to maximize tokens per second and lower tail latency on NVIDIA hardware.

Does TensorRT-LLM support multi-GPU parallelism for large model serving?▼

TensorRT-LLM supports multi-GPU parallelism through tensor parallelism, pipeline parallelism, and expert parallelism, enabling single-node and multi-node scaling of large models for production workloads.

Can I deploy an OpenAI-compatible API server for chat completions using TensorRT-LLM?▼

You can deploy an OpenAI-compatible server using the trtllm-serve command, providing monitoring endpoints and operational settings for production chat completion workloads with concurrent traffic.

What quantization formats are supported for LLM inference optimization with TensorRT-LLM?▼

TensorRT-LLM supports FP8, INT4, and FP4 quantization configurations to reduce memory usage and accelerate inference, requiring correct dtype settings to meet performance targets on compatible GPUs.

Why does TensorRT-LLM achieve faster inference than standard PyTorch serving?▼

TensorRT-LLM achieves faster inference than PyTorch by compiling model execution specifically for NVIDIA GPUs, utilizing CUDA graphs, in-flight batching, and paged KV cache to minimize latency and maximize sustained throughput.

Do I need specific NVIDIA hardware to use TensorRT-LLM for production serving?▼

TensorRT-LLM requires NVIDIA GPU compatibility for execution, with production deployments typically targeting A100 or H100 hardware to effectively leverage quantization and parallelism for real-time chat responses.

Related Skills

Explore More Software Engineering Skills→
devMoezdevMoez
1

tensorrt-llm

Compile and serve LLMs with NVIDIA TensorRT-LLM for reduced latency.

Community
Advanced
ZardLi1115ZardLi1115
4

tensorrt-llm

Compile and serve LLMs on NVIDIA GPUs with TensorRT-LLM.

Community
Intermediate
AveerAveer
1

zeus-llm

Orchestrate deployment and configuration of local LLM inference servers for OpenCode.

Community
Intermediate
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced