skills.rest
overviewlabs
overviewlabsCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference with GPU-accelerated serving and rapid cold-start times.

Updated Apr 10, 2026
Search Tags:#quantization#gpu-acceleration#serving#inference-optimization#tensorrt-llm#multigpu#production-serving
One-click install
npx skills add https://github.com/overviewlabs/WHOX --skill tensorrt-llm-overviewlabs
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/overviewlabs/WHOX/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/overviewlabs/WHOX --skill tensorrt-llm-overviewlabs

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM provides a production-grade path to maximize LLM inference throughput and minimize latency on NVIDIA GPUs, enabling scalable, low-latency AI workloads.

Core Features & Use Cases

  • High-throughput inference with FP8/INT4 quantization and CUDA optimizations
  • Multi-GPU scaling with in-flight batching for large models
  • Production-grade serving, model deployment, and monitoring support

Quick Start

Start serving by launching the trtllm-serve command with your model to enable accelerated inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs?▼

Maximize LLM inference throughput on NVIDIA GPUs by applying TensorRT-LLM with FP8/INT4 quantization and in-flight batching for low-latency production deployments.

What is needed to run multi-GPU serving for large models with TensorRT-LLM?▼

Multi-GPU serving for large models requires CUDA, TensorRT, compatible containerized deployment workflows, and multi-GPU configurations to execute TensorRT-LLM efficiently.

How do I start serving a model with TensorRT-LLM?▼

Start serving by launching the trtllm-serve command with your model to enable accelerated, high-throughput inference using TensorRT-LLM optimizations.

Can I use FP8 or INT4 quantization for large language model inference?▼

FP8 and INT4 quantization are supported for large language model inference to apply CUDA optimizations and maximize throughput on compatible NVIDIA GPU hardware.

What is the best way to scale production LLM serving across multiple GPUs?▼

Scale production LLM serving across multiple GPUs by utilizing TensorRT-LLM with in-flight batching to minimize latency and maximize throughput for large models.

What limitations exist for deploying LLMs with TensorRT-LLM?▼

Deployment limitations include strict dependencies on CUDA, TensorRT, and containerized workflows, requiring compatible models and specific multi-GPU configurations for TensorRT-LLM execution.

Related Skills

Explore More Software Engineering Skills→
m0atm0at
1

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
gagan114662gagan114662

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
ricablericable
1

@ruvector/ruvllm-cli

Run local GGUF model inference with Metal and CUDA acceleration.

Community
Intermediate
XyHalcyonXyHalcyon

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM, torch, and transformers.

Community
Advanced