skills.rest
matlee0409
matlee0409COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated Aug 23, 2026
Search Tags:#gpu#serving#fp8#inference-optimization#tensorrt#int4#multi-gpu
One-click install
npx skills add https://github.com/matlee0409/cronus --skill tensorrt-llm-matlee0409
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/matlee0409/cronus/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/matlee0409/cronus --skill tensorrt-llm-matlee0409

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

Optimizes LLM inference on NVIDIA GPUs by using TensorRT-LLM to maximize throughput and minimize latency.

Core Features & Use Cases

  • In-flight batching, quantization (FP8, INT4), and CUDA graphs for faster serving.
  • Tensor, pipeline, and expert parallelism support, plus multi-node deployments for large models.
  • Real-world scenarios include serving Llama-3 or other large models in high-throughput environments.

Quick Start

Start trtllm-serve with your model and configure tensor parallelism, batch size, and quantization to achieve production-ready throughput.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs?▼

Maximize LLM inference throughput on NVIDIA GPUs by deploying TensorRT-LLM with in-flight batching, CUDA graphs, and FP8 or INT4 quantization to minimize latency in production environments.

Can I serve large models like Llama-3 across multiple GPUs and nodes?▼

Serve large models like Llama-3 across multiple GPUs and nodes by configuring TensorRT-LLM with tensor, pipeline, and expert parallelism for multi-node scaling.

Do I need a specific GPU environment to run TensorRT-LLM for inference optimization?▼

Running TensorRT-LLM for inference optimization requires a CUDA-enabled GPU environment with TensorRT installed, plus the Python packages tensorrt-llm and torch, to execute via trtllm-serve or the Python API.

What quantization formats does TensorRT-LLM support for faster serving?▼

TensorRT-LLM supports FP8 and INT4 quantization formats for faster serving, reducing memory footprint while maintaining high throughput for large language models.

How to start serving a model using trtllm-serve with tensor parallelism?▼

Start trtllm-serve with your model and configure tensor parallelism, batch size, and quantization to achieve production-ready throughput for your specific large language model.

What is the best way to reduce latency for LLM production deployments?▼

Reduce latency for LLM production deployments by using TensorRT-LLM to apply in-flight batching, CUDA graphs, and quantization, maximizing throughput on NVIDIA GPUs.

Related Skills

Explore More Software Engineering Skills→
math-incmath-inc
1.2k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Official
Advanced
Chris-Chai-MinjaeChris-Chai-Minjae

tensorrt-llm

Accelerate large language model inference on NVIDIA GPUs with FP8/INT4 quantization.

Community
Advanced
cxnaivecxnaive

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM.

Community
Advanced
informatico-madridinformatico-madrid
1

tensorrt-llm

Optimizes LLM serving on NVIDIA hardware with batched GPU inference.

Community
Advanced