skills.rest
ruiyangruiyi
ruiyangruiyiCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Compile and serve LLMs with NVIDIA TensorRT-LLM for quantized multi-GPU inference.

Updated May 26, 2026
Search Tags:#llm inference#openai-compatible api#fp8 quantization#in-flight batching#multi-gpu parallelism#nvidia tensorRT-llm#int4 serving
One-click install
npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill tensorrt-llm-ruiyangruiyi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/ruiyangruiyi/hermes-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill tensorrt-llm-ruiyangruiyi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you optimize LLM inference to achieve maximum throughput and lowest latency by using NVIDIA TensorRT-LLM on production GPU hardware.

Core Features & Use Cases

  • High-throughput inference serving: Use in-flight batching, paged KV cache, and optimized attention kernels to raise tokens/sec.
  • Low-latency production deployment: Apply CUDA graphs and efficient batching to reduce tail latency for real-time chat and completion workloads.
  • Scalable multi-GPU and quantized serving: Enable tensor/pipeline/expert parallelism and serve quantized models (FP8/INT4/FP4) for efficient scaling across one or many GPUs.

Use case example: You run a high-traffic chat service on A100/H100 and need to serve a Llama model with quantization and in-flight batching so you can lower latency while increasing tokens/sec.

Quick Start

Start a production server for an OpenAI-compatible chat endpoint by running trtllm-serve with the target model and your tensor-parallel size (example model: meta-llama/Meta-Llama-3-8B).

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference and lower latency for a high-traffic chat service?▼

Accelerate LLM inference by compiling models with NVIDIA TensorRT-LLM to utilize in-flight batching, paged KV cache, and CUDA graphs, significantly raising tokens per second and reducing tail latency.

What is in-flight batching and how does it optimize LLM serving throughput?▼

In-flight batching optimizes LLM serving throughput by dynamically managing incoming requests during generation, allowing paged KV cache and optimized attention kernels to maximize hardware utilization and overall tokens per second.

Can I serve quantized models using FP8 or INT4 precision on multiple GPUs?▼

Yes, you can serve quantized models using FP8, INT4, or FP4 precision across multiple GPUs by enabling tensor, pipeline, or expert parallelism to efficiently scale single-node or multi-GPU deployments.

Does TensorRT-LLM support deploying an OpenAI-compatible API endpoint for production?▼

Yes, TensorRT-LLM supports deploying an OpenAI-compatible chat endpoint for production serving, providing necessary monitoring and health endpoints alongside optimized inference for real-time workloads.

What is the best way to scale Llama model inference across A100 or H100 GPUs?▼

Scale Llama model inference across A100 or H100 GPUs by applying tensor parallelism and FP8 quantization, leveraging optimized attention kernels and in-flight batching to lower latency while increasing throughput.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Deploy and optimize LLM inference servers with vLLM using OpenAI-compatible APIs and quantization.

Community
Advanced
Z43LZ43L

tensorrt-llm

Deploys TensorRT-LLM to optimize LLM inference on NVIDIA GPUs.

Community
Advanced
lxt070409lxt070409
1

serving-llms-vllm

Automate high-throughput LLM inference with vLLM and OpenAI-compatible endpoints.

Community
Advanced