skills.rest
JuanMS20
JuanMS20COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated May 3, 2026
Search Tags:#production#optimization#llm#gpu#inference#serving#tensorrt
One-click install
npx skills add https://github.com/JuanMS20/solviora-agent --skill tensorrt-llm-juanms20
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/JuanMS20/solviora-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/JuanMS20/solviora-agent --skill tensorrt-llm-juanms20

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates and optimizes large language model inference on NVIDIA GPUs, delivering higher throughput and lower latency than standard PyTorch runtimes.

Core Features & Use Cases

  • High-throughput inference: TensorRT-based optimizations enable faster serving of LLMs at scale.
  • Quantization support: FP8 and INT4 to reduce memory footprint and improve throughput with minimal accuracy loss.
  • Multi-GPU and multi-node serving: scalable deployments across GPUs and nodes with parallelism options.
  • In-flight batching: dynamic batching during inference to maximize utilization.
  • Production-grade serving: OpenAI-compatible API, built-in metrics, health checks, and deployment guidance.

Quick Start

Install the NVIDIA TensorRT-LLM package or container and start a serving instance.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs?▼

To maximize LLM inference throughput on NVIDIA GPUs, you can use TensorRT-LLM to apply in-flight batching and TensorRT-based optimizations. This enables faster serving of large language models at scale compared to standard PyTorch runtimes.

What is the best way to reduce LLM latency and memory footprint during production serving?▼

The best way to reduce LLM latency and memory footprint during production serving is applying FP8 or INT4 quantization with TensorRT-LLM. This improves throughput significantly while maintaining minimal accuracy loss on NVIDIA hardware.

Do I need specific CUDA drivers and GPU VRAM to run TensorRT-LLM deployments?▼

Yes, you need specific CUDA drivers, TensorRT, Python 3.10+, and a compatible NVIDIA GPU with sufficient VRAM to run TensorRT-LLM deployments. This setup is required to execute target models effectively on A100 or H100 hardware.

Can I scale LLM serving across multiple GPUs and nodes using TensorRT?▼

Yes, you can scale LLM serving across multiple GPUs and nodes using TensorRT-LLM. It provides multi-GPU and multi-node serving capabilities with various parallelism options to handle scalable production deployments.

How does in-flight batching work for dynamic LLM inference requests?▼

In-flight batching works for dynamic LLM inference requests by dynamically grouping requests during execution. TensorRT-LLM utilizes this technique to maximize GPU utilization and improve overall serving efficiency in production environments.

Does TensorRT-LLM provide an OpenAI-compatible API for production deployments?▼

Yes, TensorRT-LLM provides an OpenAI-compatible API for production deployments. It includes built-in metrics, health checks, and deployment guidance to ensure production-grade serving of your optimized large language models.

Related Skills

Explore More Software Engineering Skills→
math-incmath-inc
1.2k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Official
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
m0atm0at
1

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced