skills.rest
adm-humanerd
adm-humanerdCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

Updated Aug 23, 2026
Search Tags:#deployment#gpu#quantization#inference#serving#tensorrt#multi-gpu
One-click install
npx skills add https://github.com/adm-humanerd/drewgent --skill tensorrt-llm-adm-humanerd
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/adm-humanerd/drewgent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/adm-humanerd/drewgent --skill tensorrt-llm-adm-humanerd

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference on NVIDIA GPUs using TensorRT for maximum throughput and minimum latency.

Core Features & Use Cases

  • Production-grade serving with OpenAI-compatible API and automatic model compilation.
  • Supports FP8/INT4 quantization, in-flight batching, KV cache optimizations, and multi-GPU scaling for large models.
  • Useful for deployments requiring high throughput, low latency, and scalable inference across single or multi-node GPU clusters.

Quick Start

Start a production deployment with TensorRT-LLM using the recommended Docker image and configure tp_size, dtype, and max_batch_size for optimal throughput.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference on NVIDIA GPUs for maximum throughput?▼

Optimize LLM inference on NVIDIA GPUs by compiling models with TensorRT to achieve maximum throughput and minimum latency. This process enables production-grade serving, in-flight batching, and KV cache optimizations for scalable deployments.

What is the best way to deploy large models across multi-node GPU clusters?▼

The best way to deploy large models across multi-node GPU clusters is using TensorRT-LLM with multi-GPU scaling. This approach supports high-throughput, low-latency inference and automatic model compilation for production-grade serving.

Does TensorRT-LLM support FP8 and INT4 quantization for inference?▼

Yes, TensorRT-LLM supports FP8 and INT4 quantization for inference. Applying these quantization methods reduces memory footprint and increases throughput while maintaining scalable, low-latency inference on NVIDIA GPUs.

Do I need PyTorch and CUDA to run TensorRT-LLM for production serving?▼

Yes, you need PyTorch, CUDA, TensorRT, and the tensorrt_llm package to run TensorRT-LLM for production serving. These dependencies are required to enable features like in-flight batching and CUDA graphs.

How do I configure a production deployment with TensorRT-LLM?▼

Configure a production deployment with TensorRT-LLM by using the recommended Docker image and setting tp_size, dtype, and max_batch_size for optimal throughput. This setup provides an OpenAI-compatible API and automatic model compilation.

When should I use in-flight batching and CUDA graphs for LLM serving?▼

Use in-flight batching and CUDA graphs for LLM serving when you require maximum throughput and minimum latency on NVIDIA GPUs. These features optimize execution overhead and dynamic request handling in production-grade deployments.

Related Skills

Explore More Software Engineering Skills→
Monjyu1101Monjyu1101
1

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
qcmuuqcmuu

tensorrt-llm

Optimize and serve LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Intermediate
cloudlinesscloudliness

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
richardnguyen0715richardnguyen0715

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced