tensorrt-llm

Name: tensorrt-llm
Availability: InStock
Author: plaidmustache

Deploy TensorRT-LLM on NVIDIA GPUs for quantized LLM inference with batching.

Updated May 4, 2026

Search Tags:#gpu-acceleration #distributed-inference #fp8 #inference-optimization #tensorrt-llm #int4 #production-serving

Ready to run

npx skills add https://github.com/Plaidmustache/hermes-nulab --skill tensorrt-llm-plaidmustache

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Plaidmustache/hermes-nulab/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/Plaidmustache/hermes-nulab --skill tensorrt-llm-plaidmustache

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM addresses the need for production-grade, high-throughput, low-latency inference on NVIDIA GPUs by optimizing model execution and providing a robust serving interface.

Core Features & Use Cases

Fast, quantized inference (FP8/INT4) on A100/H100 for large models
In-flight batching and KV caching optimizations to maximize throughput
Multi-GPU and multi-node deployment with TensorRT-LLM serving via trtllm-serve
Use Cases: real-time chatbots, ML-assisted workflows, and large-scale AI services in production.

Quick Start

Launch trtllm-serve with a target model and basic parallelism to begin serving.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I optimize LLM inference for low latency and high throughput on NVIDIA GPUs?▼

Optimize LLM inference on NVIDIA GPUs by deploying TensorRT-LLM with FP8 or INT4 quantization, in-flight batching, and KV cache optimizations to minimize latency while maximizing throughput.

What is the best way to serve large language models in production with multi-GPU scaling?▼

Serve large language models in production using TensorRT-LLM's multi-GPU and multi-node deployment capabilities via the trtllm-serve command to achieve high-throughput scaling across connected GPUs.

Do I need A100 or H100 GPUs to use FP8 and INT4 quantization for inference?▼

Yes, FP8 and INT4 quantization for inference requires compatible NVIDIA GPUs such as A100 or H100, alongside CUDA, to execute optimized production-grade model serving.

Does TensorRT-LLM provide an OpenAI-compatible API for production serving?▼

Yes, TensorRT-LLM provides an OpenAI-compatible API for production serving, ensuring seamless integration for real-time chatbots and ML-assisted workflows.

Why use in-flight batching and KV caching for real-time LLM chatbot inference?▼

In-flight batching and KV caching maximize throughput for real-time LLM chatbot inference by dynamically managing request scheduling and reducing redundant memory allocations during generation.

When should I deploy distributed inference with multi-node environments?▼

Deploy distributed inference with multi-node environments when scaling large-scale AI services that exceed the compute or memory capacity of a single GPU node for production serving.