skills.rest
GregsGreyCode
GregsGreyCodeCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

6|2|Updated Mar 19, 2026
Search Tags:#llm#batching#quantization#inference#nvidia#tensorrt#multigpu
One-click install
npx skills add https://github.com/GregsGreyCode/Logos --skill tensorrt-llm-gregsgreycode
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/GregsGreyCode/Logos/tree/main/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/GregsGreyCode/Logos --skill tensorrt-llm-gregsgreycode

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM reduces latency and increases throughput for large language model inference on NVIDIA GPUs by leveraging TensorRT optimizations, enabling production-grade deployment on GPUs such as A100 and H100. It supports quantization (FP8/INT4), in-flight batching, and multi-GPU scaling to meet real-time or high-throughput requirements.

Core Features & Use Cases

  • Performance optimizations: In-flight batching, paged KV cache, CUDA graphs, and quantization for speedups and reduced memory footprint.
  • Parallelism & scaling: Tensor parallelism, pipeline parallelism, expert parallelism for Mixture-of-Experts models, and multi-node deployments for very large models.
  • Production-ready serving: OpenAI-compatible API, automatic model download/compilation, metrics, health checks, and deployment patterns for robust inference services.

Quick Start

Start a TensorRT-LLM deployment on NVIDIA GPUs to serve an LLM with FP8 quantization and multi-GPU parallelism.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce LLM inference latency on NVIDIA GPUs for production?▼

LLM inference latency on NVIDIA GPUs is reduced by leveraging TensorRT optimizations like in-flight batching, paged KV cache, and CUDA graphs. This approach enables production-grade deployment on A100 and H100 GPUs to meet real-time requirements.

Does TensorRT support FP8 and INT4 quantization for large language models?▼

FP8 and INT4 quantization are supported to accelerate large language model inference and reduce memory footprint. These TensorRT optimizations decrease latency and increase throughput for production deployments on compatible NVIDIA GPUs.

What is the best way to scale multi-GPU LLM inference across clusters?▼

Multi-GPU LLM inference scaling is achieved using tensor parallelism, pipeline parallelism, and expert parallelism for Mixture-of-Experts models. This supports multi-node deployments for very large models requiring maximum throughput.

Can I use PyTorch models with TensorRT for production LLM serving?▼

Compatible PyTorch models are required to use TensorRT for production LLM serving. The system provides an OpenAI-compatible API, automatic model download and compilation, metrics, and health checks for robust inference services.

Do I need specific hardware drivers for TensorRT LLM inference?▼

CUDA-enabled hardware with NVIDIA drivers and the TensorRT runtime are required for LLM inference. This setup ensures you can leverage quantization, in-flight batching, and multi-GPU scaling on GPUs like A100 and H100.

Related Skills

Explore More Software Engineering Skills→
arsityarsity

tensorrt-llm

Optimize LLM inference throughput and latency on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
OpenLAIROpenLAIR
1.0k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and multi-GPU scaling.

Official
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced