skills.rest
cloudliness
cloudlinessCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated Mar 26, 2026
Search Tags:#quantization#gpu-acceleration#inference-serving#fp8#llm-inference#tensorrt#multi-gpu
One-click install
npx skills add https://github.com/cloudliness/Hermes-Autonomous-AI-Agent-Dialed-In-For-Windows-11 --skill tensorrt-llm-cloudliness
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/cloudliness/Hermes-Autonomous-AI-Agent-Dialed-In-For-Windows-11/tree/main/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/cloudliness/Hermes-Autonomous-AI-Agent-Dialed-In-For-Windows-11 --skill tensorrt-llm-cloudliness

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference on NVIDIA GPUs by leveraging TensorRT to maximize throughput and minimize latency in production deployments.

Core Features & Use Cases

  • Quantization support (FP8/INT4) to reduce memory footprint and increase speed for large models.
  • In-flight batching and efficient KV cache management to maximize throughput.
  • Multi-GPU and multi-node deployment with tensor, pipeline, and expert parallelism for very large models.
  • CUDA graphs and optimized kernels to lower kernel launch overhead and latency.
  • Production serving scenarios including high-throughput chat/completion endpoints and model offload strategies.

Quick Start

Install tensorrt-llm, point it at your model, and start the trtllm-serve endpoint to begin serving locally.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs?▼

To maximize LLM inference throughput on NVIDIA GPUs, use TensorRT to apply FP8 or INT4 quantization, in-flight batching, and paged KV cache management. This minimizes latency and increases processing speed for production deployments of large models.

Does TensorRT support FP8 quantization and in-flight batching for large models?▼

Yes, TensorRT supports FP8 and INT4 quantization to reduce memory footprint, alongside in-flight batching and efficient paged KV cache management. These features work together to maximize throughput and minimize latency during large model inference.

How do I deploy LLMs across multiple GPUs and nodes?▼

Deploy LLMs across multiple GPUs and nodes by utilizing tensor, pipeline, and expert parallelism strategies. This approach enables very large models to be distributed efficiently, while CUDA graphs and optimized kernels lower kernel launch overhead and latency.

What is the best way to reduce memory footprint during LLM inference serving?▼

The best way to reduce memory footprint during LLM inference serving is applying FP8 or INT4 quantization. Combined with paged KV cache management, this significantly reduces memory requirements while maintaining high throughput for production model serving.

Can I use TensorRT for high-throughput chat and completion endpoints in production?▼

Yes, TensorRT is designed for production serving scenarios including high-throughput chat and completion endpoints. It leverages in-flight batching, CUDA graphs, and model offload strategies to effectively serve large models in production environments.

How do I start serving an LLM locally using TensorRT?▼

Start serving an LLM locally by installing TensorRT, pointing it at your model, and starting the trtllm-serve endpoint. This initiates local inference serving with optimized kernels and efficient KV cache management.

Related Skills

Explore More Software Engineering Skills→
DarkArty07DarkArty07
3

llama-cpp

Deploy local LLM inference with GGUF quantized weights via llama.cpp.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
PiaoxuemoliPiaoxuemoli

serving-llms-vllm

Serve LLMs with OpenAI-compatible APIs using vLLM and quantization.

Community
Advanced