skills.rest
Plaidmustache
PlaidmustacheCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Deploy TensorRT-LLM on NVIDIA GPUs for quantized LLM inference with batching.

Updated May 4, 2026
Search Tags:#gpu-acceleration#distributed-inference#fp8#inference-optimization#tensorrt-llm#int4#production-serving
One-click install
npx skills add https://github.com/Plaidmustache/hermes-nulab --skill tensorrt-llm-plaidmustache
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Plaidmustache/hermes-nulab/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/Plaidmustache/hermes-nulab --skill tensorrt-llm-plaidmustache

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM addresses the need for production-grade, high-throughput, low-latency inference on NVIDIA GPUs by optimizing model execution and providing a robust serving interface.

Core Features & Use Cases

  • Fast, quantized inference (FP8/INT4) on A100/H100 for large models
  • In-flight batching and KV caching optimizations to maximize throughput
  • Multi-GPU and multi-node deployment with TensorRT-LLM serving via trtllm-serve
  • Use Cases: real-time chatbots, ML-assisted workflows, and large-scale AI services in production.

Quick Start

Launch trtllm-serve with a target model and basic parallelism to begin serving.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for low latency and high throughput on NVIDIA GPUs?▼

Optimize LLM inference on NVIDIA GPUs by deploying TensorRT-LLM with FP8 or INT4 quantization, in-flight batching, and KV cache optimizations to minimize latency while maximizing throughput.

What is the best way to serve large language models in production with multi-GPU scaling?▼

Serve large language models in production using TensorRT-LLM's multi-GPU and multi-node deployment capabilities via the trtllm-serve command to achieve high-throughput scaling across connected GPUs.

Do I need A100 or H100 GPUs to use FP8 and INT4 quantization for inference?▼

Yes, FP8 and INT4 quantization for inference requires compatible NVIDIA GPUs such as A100 or H100, alongside CUDA, to execute optimized production-grade model serving.

Does TensorRT-LLM provide an OpenAI-compatible API for production serving?▼

Yes, TensorRT-LLM provides an OpenAI-compatible API for production serving, ensuring seamless integration for real-time chatbots and ML-assisted workflows.

Why use in-flight batching and KV caching for real-time LLM chatbot inference?▼

In-flight batching and KV caching maximize throughput for real-time LLM chatbot inference by dynamically managing request scheduling and reducing redundant memory allocations during generation.

When should I deploy distributed inference with multi-node environments?▼

Deploy distributed inference with multi-node environments when scaling large-scale AI services that exceed the compute or memory capacity of a single GPU node for production serving.

Related Skills

Explore More Software Engineering Skills→
m0atm0at
1

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
framerslabframerslab

streaming-stt-whisper

Transcribe live speech in real time using the OpenAI Whisper API.

Official
Intermediate
Chris-Chai-MinjaeChris-Chai-Minjae

tensorrt-llm

Accelerate large language model inference on NVIDIA GPUs with FP8/INT4 quantization.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced