skills.rest
box755
box755COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM runtime.

Updated Aug 27, 2026
Search Tags:#quantization#low latency#model serving#tensorrt#llm inference#multi-gpu#nvidia gpus
One-click install
npx skills add https://github.com/box755/simlens-research --skill tensorrt-llm-box755
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/box755/simlens-research/tree/main/skill-packs/AI-Research-SKILLs/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/box755/simlens-research --skill tensorrt-llm-box755

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM solves slow, inefficient large language model inference by compiling and optimizing execution on NVIDIA GPUs to achieve much lower latency and higher throughput than typical PyTorch serving.

Core Features & Use Cases

  • Production-grade LLM serving: Run an OpenAI-compatible server via trtllm-serve with batching and scalable deployment patterns for real-time chat/completions.
  • Inference optimization for NVIDIA GPUs: Apply TensorRT-LLM kernel optimizations (including Flash Attention, CUDA graphs, and KV cache techniques) to reduce end-to-end time per token.
  • Quantization and parallelism for larger models: Use FP8/INT4/F P4 quantization and scale across GPUs with tensor/pipeline/expert parallelism to fit and accelerate bigger model classes.

Quick Start

Start serving a quantized Llama model on 4 GPUs by running: trtllm-serve meta-llama/Meta-Llama-3-8B --tp_size 4 --max_batch_size 256 --max_num_tokens 4096.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for lower latency on NVIDIA GPUs?▼

LLM inference latency on NVIDIA GPUs is reduced by applying TensorRT-LLM kernel optimizations like Flash Attention, CUDA graphs, and KV cache techniques to increase tokens-per-second for production serving workflows.

What's the best way to serve a quantized LLM across multiple GPUs?▼

Serving a quantized LLM across multiple GPUs is done using tensor, pipeline, or expert parallelism combined with FP8, INT4, or FP4 quantization to fit and accelerate larger model classes during inference.

How do I run an OpenAI-compatible LLM server with dynamic batching?▼

An OpenAI-compatible LLM server with dynamic batching is run using the trtllm-serve command, configuring max_batch_size and max_num_tokens to handle real-time chat and completion endpoints at scale.

Does TensorRT-LLM work with PyTorch models for production serving?▼

TensorRT-LLM works with PyTorch models by compiling and optimizing their execution on NVIDIA GPUs, replacing typical PyTorch serving to achieve much higher throughput for production workloads.

When should I use FP8 or INT4 quantization for LLM deployment?▼

FP8 or INT4 quantization for LLM deployment should be used when you need to fit and accelerate bigger model classes on NVIDIA GPUs, reducing memory footprint and increasing inference throughput.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
quiznatquiznat

serving-llms-vllm

Deploy vLLM servers with PagedAttention, continuous batching, and OpenAI-compatible endpoints.

Community
Intermediate
mindspore-aimindspore-ai
258

triton-cuda-attention

Implement Triton-CUDA attention kernels with online Softmax for transformer workloads.

Official
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced