skills.rest
jleechanorg
jleechanorgOFFICIAL
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Updated Jun 28, 2026
Search Tags:#optimization#llm#inference#serving#tensorrt#multi-gpu
One-click install
npx skills add https://github.com/jleechanorg/hermes-agent --skill tensorrt-llm-jleechanorg
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/jleechanorg/hermes-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/jleechanorg/hermes-agent --skill tensorrt-llm-jleechanorg

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM speeds up large language model inference on NVIDIA GPUs, enabling production-grade deployments with lower latency and higher throughput.

Core Features & Use Cases

  • In-flight batching and KV cache optimization for maximum throughput.
  • Multi-GPU and multi-node deployment using tensor and pipeline parallelism strategies.
  • Production-ready serving with trtllm-serve and OpenAI-compatible API support.
  • Quantization options (FP8, INT4) to reduce memory footprint and boost speed across devices.

Quick Start

Install Tensorrt-LLM and run trtllm-serve with your model to start serving.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for lower latency and higher throughput on NVIDIA GPUs?▼

To optimize LLM inference on NVIDIA GPUs, you can use TensorRT for high throughput and low latency. It targets production deployment with in-flight batching, KV cache optimization, and multi-GPU scaling.

Does TensorRT support multi-GPU and multi-node scaling for LLM serving?▼

Yes, TensorRT supports multi-GPU and multi-node scaling for LLM serving using tensor and pipeline parallelism strategies. This allows you to distribute inference workloads across multiple NVIDIA GPUs for improved performance.

How do I reduce memory footprint during LLM inference?▼

You can reduce memory footprint during LLM inference using quantization options like FP8 and INT4 provided by TensorRT. These options help boost speed across devices while lowering memory requirements.

Can I serve LLMs with an OpenAI-compatible API using TensorRT?▼

Yes, you can serve LLMs with an OpenAI-compatible API using trtllm-serve. This provides production-ready serving capabilities, allowing you to start serving your models after installing TensorRT-LLM.

What do I need to run TensorRT-LLM for production deployment?▼

To run TensorRT-LLM for production deployment, you need CUDA-enabled GPUs and a compatible software stack including tensorrt_llm and torch. This environment enables high throughput and low latency LLM inference.

What is in-flight batching and how does it improve LLM inference?▼

In-flight batching is a technique used in TensorRT-LLM to optimize KV cache and maximize throughput during LLM inference. It dynamically manages incoming requests to reduce latency and improve overall serving efficiency.

Related Skills

Explore More Software Engineering Skills→
KappTech88KappTech88

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
gagan114662gagan114662

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
m0atm0at
1

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced