skills.rest
LynxLabVN
LynxLabVNOFFICIAL
Explore all skills by this author →
GitHub

tensorrt-llm

Accelerate LLM inference on NVIDIA GPUs using TensorRT.

Updated Jul 3, 2026
Search Tags:#performance#LLM#GPU#inference#TensorRT#NVIDIA
One-click install
npx skills add https://github.com/LynxLabVN/office-agent --skill tensorrt-llm-lynxlabvn
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/LynxLabVN/office-agent/tree/main/agent-core/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/LynxLabVN/office-agent --skill tensorrt-llm-lynxlabvn

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill addresses the challenge of slow LLM inference on NVIDIA GPUs, providing a solution that achieves high throughput and low latency for production deployment.

Core Features & Use Cases

  • Optimized Inference: Accelerates LLM inference using NVIDIA TensorRT, achieving 10-100x faster performance than PyTorch.
  • Quantization Support: Handles quantization for models with FP8/INT4 precision, reducing memory usage and improving throughput.
  • Multi-GPU Scaling: Supports multi-GPU scaling for serving models across multiple GPUs or nodes.
  • Use Case: Ideal for serving large LLM models in production environments, especially on NVIDIA A100/H100 GPUs, where high performance is critical.

Quick Start

To get started, run the following command:

docker pull nvidia/tensorrt_llm:latest

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production deployment?▼

Accelerate LLM inference on NVIDIA GPUs using TensorRT to achieve high throughput and low latency for production deployment. It optimizes model execution to run 10-100x faster than standard PyTorch.

Can I use TensorRT to quantize LLM models with FP8 or INT4 precision?▼

Yes, TensorRT supports LLM quantization using FP8 and INT4 precision. This reduces memory usage and significantly improves inference throughput on supported NVIDIA hardware.

How do I scale LLM inference across multiple NVIDIA A100 or H100 GPUs?▼

Scale LLM inference across multiple NVIDIA A100 or H100 GPUs using TensorRT multi-GPU scaling. This allows you to serve large models distributed across multiple GPUs or nodes efficiently.

What is the performance difference between TensorRT and PyTorch for LLM inference?▼

TensorRT achieves 10-100x faster LLM inference performance compared to PyTorch. It optimizes execution graphs specifically for NVIDIA GPUs to maximize throughput and minimize latency.

Do I need Docker to get started with TensorRT for LLM inference?▼

You can pull the nvidia/tensorrt_llm Docker image to quickly set up your environment for LLM inference. This provides a pre-configured container with the necessary dependencies installed.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
hung-phanhung-phan
1

triton-inference-server

Serve ML models with dynamic batching and concurrent execution via Triton Inference Server.

Community
Advanced
informatico-madridinformatico-madrid
1

tensorrt-llm

Optimizes LLM serving on NVIDIA hardware with batched GPU inference.

Community
Advanced