skills.rest
ever-oli
ever-oliCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and CUDA graphs.

3|Updated Mar 20, 2026
Search Tags:#gpu#batching#quantization#inference#serving#tensorrt-llm#multi-gpu
One-click install
npx skills add https://github.com/ever-oli/io --skill tensorrt-llm-ever-oli
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/ever-oli/io/tree/main/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/ever-oli/io --skill tensorrt-llm-ever-oli

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM provides high-throughput, low-latency LLM inference on NVIDIA GPUs by optimizing runtimes with TensorRT, enabling scalable production deployments.

Core Features & Use Cases

  • In-flight batching, quantization (FP8, INT4), multi-GPU support, and CUDA graphs for performance.
  • Production-grade serving for chat, embeddings, and real-time inference across enterprises.
  • Easy deployment across single-node to multi-node clusters with tensor and pipeline parallelism.

Quick Start

Install the package and start trtllm-serve with a chosen model and TP/PP configuration to launch an optimized inference service.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference throughput on NVIDIA GPUs?▼

Optimize LLM inference throughput on NVIDIA GPUs by applying TensorRT-LLM techniques like in-flight batching, KV cache optimization, and CUDA graphs to achieve low-latency serving.

What is the best way to deploy multi-GPU serving for large language models?▼

Deploy multi-GPU serving for large language models using tensor and pipeline parallelism across single-node or multi-node clusters to enable scalable, production-grade inference.

Can I use FP8 and INT4 quantization for TensorRT inference?▼

Yes, you can apply FP8 and INT4 quantization within TensorRT inference workflows to reduce model footprint and accelerate computation while maintaining high throughput.

Do I need CUDA toolkit and TensorRT to run production LLM serving?▼

Yes, production LLM serving with TensorRT-LLM strictly requires the CUDA toolkit, TensorRT runtime, and NVIDIA GPUs to execute optimized inference workloads.

How does in-flight batching improve real-time LLM serving performance?▼

In-flight batching improves real-time LLM serving performance by dynamically managing continuous request streams, significantly increasing GPU utilization and reducing latency.

How do I start an optimized inference service with trtllm-serve?▼

Start an optimized inference service by installing the package and running trtllm-serve with a chosen model and configured tensor or pipeline parallelism settings.

Related Skills

Explore More Software Engineering Skills→
GregsGreyCodeGregsGreyCode
6

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced
Devsoul2026Devsoul2026
150

tensorrt-llm

Optimize and deploy LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
OpenLAIROpenLAIR
1.0k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and multi-GPU scaling.

Official
Advanced
JamesFincherJamesFincher

tensorrt-llm

Optimizes LLM inference serving on NVIDIA GPUs using TensorRT-LLM with quantization and parallelism options.

Community
Advanced