skills.rest
AVOI-CEO
AVOI-CEOCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Updated Aug 23, 2026
Search Tags:#quantization#gpu-acceleration#fp8#llm-inference#tensorrt#int4
One-click install
npx skills add https://github.com/AVOI-CEO/avoi-agent --skill tensorrt-llm-avoi-ceo
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/AVOI-CEO/avoi-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/AVOI-CEO/avoi-agent --skill tensorrt-llm-avoi-ceo

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates large language model inference on NVIDIA GPUs by leveraging TensorRT optimizations to achieve higher throughput and lower latency in production environments.

Core Features & Use Cases

  • Inference acceleration through FP8/INT4 quantization, in-flight batching, and multi-GPU scaling for large models.
  • Production serving with OpenAI-compatible API, automatic model compilation, and scalable deployment for single-node and multi-node clusters.
  • Use Case: Deploy a 70B LLaMA model across 4 GPUs to reach tens of thousands of tokens per second for real-time chat workloads.

Quick Start

Run trtllm-serve with your model to automatically download, compile, and serve with TensorRT optimizations.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production?▼

Accelerate LLM inference on NVIDIA GPUs using TensorRT to achieve higher throughput and lower latency. It applies FP8/INT4 quantization, in-flight batching, and multi-GPU scaling to optimize large language model serving in production environments.

What is the best way to deploy a 70B LLaMA model across multiple GPUs?▼

Deploy a 70B LLaMA model across multiple GPUs using TensorRT optimizations to reach tens of thousands of tokens per second. It supports multi-GPU scaling across single-node and multi-node clusters for real-time chat workloads.

Do I need specific hardware to run TensorRT-LLM optimizations?▼

TensorRT-LLM optimization requires CUDA-enabled NVIDIA GPUs, NVIDIA TensorRT, and PyTorch as the runtime to function. These dependencies are necessary to leverage FP8/INT4 quantization and multi-GPU scaling for inference acceleration.

How does in-flight batching improve LLM inference throughput?▼

In-flight batching improves LLM inference throughput by dynamically scheduling requests on NVIDIA GPUs using TensorRT. Combined with FP8/INT4 quantization, it maximizes token generation rates for production serving with an OpenAI-compatible API.

Can I serve optimized LLMs with an OpenAI-compatible API?▼

Serve optimized LLMs with an OpenAI-compatible API using the trtllm-serve command. It automatically downloads, compiles, and serves your model with TensorRT optimizations for scalable production deployment.

What are the limitations of FP8 and INT4 quantization for LLM inference?▼

FP8 and INT4 quantization reduce model precision to accelerate LLM inference on NVIDIA GPUs via TensorRT. While they significantly increase throughput and lower latency, they require CUDA-enabled GPUs and may impact output quality depending on the model.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
Z43LZ43L

tensorrt-llm

Deploys TensorRT-LLM to optimize LLM inference on NVIDIA GPUs.

Community
Advanced
arsityarsity

tensorrt-llm

Optimize LLM inference throughput and latency on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
m0atm0at
1

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced