skills.rest
peteromallet
peteromalletCOMMUNITY
Explore all skills by this author →
𝕏@peteromalletGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

97|8|Updated Mar 20, 2026
Search Tags:#latency#gpu#throughput#quantization#inference#tensorrt-llm#multi-gpu
One-click install
npx skills add https://github.com/peteromallet/megaplan --skill tensorrt-llm-peteromallet
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/peteromallet/megaplan/tree/main/megaplan/agent/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/peteromallet/megaplan --skill tensorrt-llm-peteromallet

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM optimizes LLM inference on NVIDIA GPUs to deliver production-grade throughput and minimal latency for real-time and batch workloads.

Core Features & Use Cases

  • In-flight batching, CUDA graphs, and KV cache optimizations to maximize throughput.
  • FP8/INT4 quantization and multi-GPU/multi-node deployment for large models with efficient resource usage.
  • Production-serving capabilities with an OpenAI-compatible API and observability (metrics, health checks).

Quick Start

Install and run a FP8-quantized LLM on NVIDIA GPUs with trtllm-serve to validate throughput and latency.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference latency on NVIDIA GPUs?▼

You can optimize LLM inference latency on NVIDIA GPUs by applying TensorRT-LLM, which uses in-flight batching, CUDA graphs, and KV cache optimizations to maximize throughput and minimize response times in production settings.

What is the best way to deploy large models across multi-GPU and multi-node environments?▼

Deploying large models across multi-GPU and multi-node environments is handled by TensorRT-LLM through advanced parallelism configurations and FP8/INT4 quantization, ensuring efficient resource usage for large-scale inference workloads.

Can I use FP8 or INT4 quantization for LLM inference?▼

Yes, you can use FP8 and INT4 quantization for LLM inference. TensorRT-LLM supports these quantization formats to reduce memory consumption and accelerate inference speeds on CUDA-enabled NVIDIA GPUs.

Do I need a specific Python environment to run TensorRT-LLM?▼

Yes, you need a Python environment with PyTorch and TensorRT-LLM installed, along with a CUDA-enabled NVIDIA GPU. Server tooling such as trtllm-serve is also required for deployment and monitoring.

How does in-flight batching improve LLM throughput?▼

In-flight batching improves LLM throughput by dynamically managing requests during generation, which maximizes GPU utilization and reduces latency compared to static batching methods in production workloads.

Does TensorRT-LLM provide an OpenAI-compatible API for production serving?▼

Yes, TensorRT-LLM provides production-serving capabilities with an OpenAI-compatible API, along with observability features like metrics and health checks to monitor deployment performance effectively.

Related Skills

Explore More Software Engineering Skills→
GregsGreyCodeGregsGreyCode
6

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced
gagan114662gagan114662

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
heysuhasheysuhas
2

serving-llms-vllm

Deploy LLM inference servers with the vLLM engine and OpenAI-compatible APIs.

Community
Advanced
ever-oliever-oli
3

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and CUDA graphs.

Community
Advanced