skills.rest
ZardLi1115
ZardLi1115COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Compile and serve LLMs on NVIDIA GPUs with TensorRT-LLM.

4|Updated May 18, 2026
Search Tags:#inference optimization#llm serving#tensorrt-llm#nvidia gpu#multi-gpu#openai-compatible api#fp8 quantization
One-click install
npx skills add https://github.com/ZardLi1115/zedclaw --skill tensorrt-llm-zardli1115
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/ZardLi1115/zedclaw/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/ZardLi1115/zedclaw --skill tensorrt-llm-zardli1115

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM solves the problem of slow, high-latency LLM inference by optimizing model execution for NVIDIA GPUs to achieve maximum throughput.

Core Features & Use Cases

  • High-throughput inference serving: Uses TensorRT kernels with in-flight batching and efficient KV-cache handling for production workloads.
  • Low-latency scaling: Supports tensor/pipeline/expert parallelism across multiple GPUs and nodes to meet strict latency and throughput targets.
  • Quantized deployment: Enables FP8 and INT4/INT variants to reduce memory footprint and cost while maintaining strong performance.

Example use case: Serving a Llama 3 model behind an OpenAI-compatible endpoint where you need high tokens/sec, predictable latency, and support for FP8 quantization on A100/H100-class GPUs.

Quick Start

Start a production server by running TensorRT-LLM's trtllm-serve for your model with tensor parallelism, an appropriate dtype (e.g., fp8), and batching limits.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for high throughput on NVIDIA GPUs?▼

Optimize LLM inference on NVIDIA GPUs by compiling models with TensorRT-LLM to leverage TensorRT kernels, in-flight batching, and efficient KV-cache handling for maximum throughput.

Can I serve an OpenAI-compatible API with FP8 quantization on A100 or H100 GPUs?▼

Yes, serving with FP8 quantization on A100 and H100 GPUs is supported via the trtllm-serve API, providing an OpenAI-compatible endpoint with reduced memory footprint and low latency.

What's the best way to scale large language model serving across multiple GPUs?▼

Scale large language model serving across multiple GPUs by configuring tensor, pipeline, or expert parallelism in TensorRT-LLM to meet strict latency and throughput targets across nodes.

Does TensorRT-LLM support INT4 quantization to reduce model memory footprint?▼

Yes, TensorRT-LLM supports INT4 and INT quantization variants alongside FP8 to reduce memory footprint and cost while maintaining strong performance during deployment.

Why does LLM inference latency increase under concurrent production workloads?▼

LLM inference latency often increases under concurrent workloads due to inefficient batching and KV-cache management, which TensorRT-LLM solves using in-flight batching and optimized kernels.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
qcmuuqcmuu

tensorrt-llm

Optimize and serve LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Intermediate
zeronx798zeronx798

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM on A100 and H100 GPUs.

Community
Advanced
davpatel605-beepdavpatel605-beep

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced