skills.rest
math-inc
math-incOFFICIAL
Explore all skills by this author →
Website𝕏@mathematics_incGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

1.2k|116|Updated Mar 19, 2026
Search Tags:#gpu#nvidia#inference-optimization#tensorrt-llm#multi-gpu#fp8-quantization#production-serving
One-click install
npx skills add https://github.com/math-inc/OpenGauss --skill tensorrt-llm-math-inc
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/math-inc/OpenGauss/tree/main/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/math-inc/OpenGauss --skill tensorrt-llm-math-inc

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM optimizes LLM inference with TensorRT, delivering production-grade performance on NVIDIA GPUs.

Core Features & Use Cases

  • High-throughput inference: Leverage TensorRT to maximize tokens per second on A100/H100 GPUs.
  • Low-latency serving: In-flight batching and efficient memory usage for responsive deployments.
  • Multi-GPU scaling: Support TP/PP across 1–multi-node GPU clusters for large models.
  • Quantization support: FP8 and INT4 to reduce memory and increase speed with controlled accuracy loss.
  • Production-grade deployment: Ready for real-world serving with standard APIs and monitoring.

Quick Start

Launch a production-ready TensorRT-LLM server with a configured model and multi-GPU setup.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference on NVIDIA GPUs for production serving?▼

To optimize LLM inference on NVIDIA GPUs, you can use TensorRT to maximize throughput and minimize latency. It leverages in-flight batching and efficient memory management for responsive production deployments on A100 and H100 GPUs.

What is the best way to scale large models across multi-node GPU clusters?▼

The best way to scale large models across multi-node GPU clusters is using multi-GPU scaling with TensorRT. It supports tensor and pipeline parallelism across single-node and multi-node clusters to efficiently distribute large model computations.

Does TensorRT-LLM support FP8 and INT4 quantization?▼

Yes, TensorRT-LLM supports FP8 and INT4 quantization. These techniques reduce memory usage and increase inference speed with controlled accuracy loss, making it suitable for high-throughput LLM serving on compatible NVIDIA GPUs.

Do I need a specific Python environment to run TensorRT-LLM?▼

Yes, you need a specific Python environment to run TensorRT-LLM. It requires a CUDA-enabled NVIDIA GPU with supported drivers and a Python environment containing the tensorrt-llm and torch dependencies.

How do I start a production-ready LLM server with multi-GPU setup?▼

To start a production-ready LLM server with a multi-GPU setup, use the TensorRT-LLM quick start configuration. It enables rapid deployment of standard APIs and monitoring for real-world serving on NVIDIA GPUs.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
richardnguyen0715richardnguyen0715

serving-llms-vllm

Deploy and serve LLMs with vLLM, OpenAI-compatible endpoints, and quantization.

Community
Advanced
NVIDIANVIDIA
13.2k

trt-onnx-quickstart

Convert ONNX models to TensorRT engines and verify numerical parity.

Official
Advanced
ldzhhxxldzhhxx
1

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for production GPUs.

Community
Advanced