skills.rest
arsity
arsityCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Optimize LLM inference throughput and latency on NVIDIA GPUs with TensorRT-LLM.

Updated Mar 16, 2026
Search Tags:#production#quantization#inference#gpu-acceleration#serving#tensorrt-llm#multi-gpu
One-click install
npx skills add https://github.com/arsity/scholar-tools --skill tensorrt-llm-arsity
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/arsity/scholar-tools/tree/main/vendor/ai-research-skills/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/arsity/scholar-tools --skill tensorrt-llm-arsity

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates large language model inference on NVIDIA GPUs to deliver higher throughput and lower latency in production environments.

Core Features & Use Cases

  • In-flight batching, CUDA-optimized kernels, and FP8/INT4 quantization for faster inference.
  • Multi-GPU scaling with tensor, pipeline, and expert parallelism for very large models.
  • Production-ready serving with OpenAI-compatible APIs or custom Python interfaces.

Quick Start

Run trtllm-serve with your model and a suitable tensor-parallel setup to start serving in production.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference and reduce latency on NVIDIA GPUs?▼

To accelerate LLM inference on NVIDIA GPUs, apply CUDA-optimized kernels, in-flight batching, and FP8 or INT4 quantization to significantly reduce production latency and increase throughput.

How do I serve LLMs in production with an OpenAI-compatible API?▼

Serve LLMs in production with an OpenAI-compatible API by running the trtllm-serve command with your model and a suitable tensor-parallel setup for immediate deployment.

Can I scale very large models across multiple GPUs and nodes?▼

Scale very large models across multiple GPUs and nodes using tensor, pipeline, and expert parallelism to distribute computation and overcome single-GPU memory limits.

What hardware and dependencies do I need to run TensorRT-LLM for production serving?▼

Running production serving requires CUDA-enabled NVIDIA hardware, supported PyTorch models, and the TensorRT runtime with appropriate drivers and libraries.

When should I use FP8 or INT4 quantization for LLM inference?▼

Use FP8 or INT4 quantization for LLM inference when you need to maximize throughput and minimize memory footprint in production environments without significant accuracy degradation.

Related Skills

Explore More Software Engineering Skills→
GregsGreyCodeGregsGreyCode
6

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced
cryptopaficryptopafi

serving-llms-vllm

Serve large language models with high-throughput inference for OpenAI-compatible endpoints.

Community
Advanced
PlaidmustachePlaidmustache

tensorrt-llm

Deploy TensorRT-LLM on NVIDIA GPUs for quantized LLM inference with batching.

Community
Advanced
BermudaLocalsBermudaLocals
1

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs using quantization and multi-GPU scaling.

Community
Advanced