skills.rest
OpenLAIR
OpenLAIROFFICIAL
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and multi-GPU scaling.

1.0k|117|Updated Feb 26, 2026
Search Tags:#production#quantization#inference#gpu-acceleration#llm-serving#tensorrt#multigpu
One-click install
npx skills add https://github.com/OpenLAIR/dr-claw --skill tensorrt-llm-openlair
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/OpenLAIR/dr-claw/tree/main/skills/inference-serving/tensorrt-llm
Command: npx skills add https://github.com/OpenLAIR/dr-claw --skill tensorrt-llm-openlair

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference performance on NVIDIA GPUs to maximize throughput and minimize latency in production deployments.

Core Features & Use Cases

  • Hardware-accelerated inference using TensorRT for large language models.
  • Production-grade serving with multi-GPU scaling, quantization (FP8/INT4), and in-flight batching.
  • Flexible deployment supports single-node and multi-node configurations for enterprise workloads.

Quick Start

Deploy TensorRT-LLM on your NVIDIA GPU cluster for accelerated inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs?▼

To maximize LLM inference throughput on NVIDIA GPUs, apply TensorRT hardware acceleration with in-flight batching and paged KV cache. This minimizes latency and optimizes production deployments requiring high-volume tensor parallelism.

What is the best way to scale LLM serving across multiple GPUs?▼

The best way to scale LLM serving across multiple GPUs is configuring multi-node setups with tensor, pipeline, and expert parallelism. This distributes inference efficiently across available hardware to handle enterprise workloads.

Can I use FP8 or INT4 quantization for production LLM serving?▼

Yes, you can use FP8 or INT4 quantization for production LLM serving. These quantization methods reduce memory footprint and accelerate inference on NVIDIA GPUs while maintaining acceptable model accuracy thresholds.

Does TensorRT-LLM support OpenAI-compatible API endpoints?▼

Yes, TensorRT-LLM supports OpenAI-compatible API endpoints through the trtllm-serve feature. This allows integration of optimized GPU inference into existing application pipelines without modifying API request structures.

When do I need CUDA graphs and paged KV cache for LLM inference?▼

You need CUDA graphs and paged KV cache for LLM inference when optimizing high-throughput production environments. These features reduce CPU overhead and manage memory efficiently during dynamic batching operations on NVIDIA GPUs.

How do I deploy multi-node configurations for enterprise LLM workloads?▼

To deploy multi-node configurations for enterprise LLM workloads, configure TensorRT to orchestrate multi-GPU scaling and pipeline parallelism. This enables flexible deployment across distributed clusters for demanding production inference tasks.

Related Skills

Explore More Software Engineering Skills→
FutureAtomsFutureAtoms
1

personaplex-setup

Configure PersonaPlex environments for local or cloud deployment with GPU and dependency setup.

Community
Advanced
gagan114662gagan114662

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
zeronx798zeronx798

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM on A100 and H100 GPUs.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced