skills.rest
t2ance
t2anceCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Deploy high-throughput LLM inference with TensorRT-LLM on NVIDIA GPUs.

Updated Aug 23, 2026
Search Tags:#production#llm#gpu#quantization#inference#serving#tensorrt
One-click install
npx skills add https://github.com/t2ance/dr-claw-plugin --skill tensorrt-llm-t2ance
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/t2ance/dr-claw-plugin/tree/main/plugins/ml-training-stack/skills/inference-serving/tensorrt-llm
Command: npx skills add https://github.com/t2ance/dr-claw-plugin --skill tensorrt-llm-t2ance

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates production-grade LLM inference on NVIDIA GPUs, delivering high throughput and low latency for enterprise deployments.

Core Features & Use Cases

  • High-throughput inference with TensorRT optimizations (FP8/INT4, flash attention, KV cache tuning)
  • Multi-GPU and multi-node scaling for very large models
  • Production-ready serving with OpenAI-compatible API and Prometheus metrics

Quick Start

Install and run the trtllm-serve server with a model to begin production inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy high-throughput LLM inference on NVIDIA GPUs?▼

Deploy high-throughput LLM inference on NVIDIA GPUs using TensorRT-LLM to optimize model loading, configure server parallelism, and enable in-flight batching for low-latency enterprise production.

Can I use FP8 or INT4 quantization for production LLM serving?▼

Yes, you can use FP8 or INT4 quantization for production LLM serving. TensorRT-LLM supports these quantization methods alongside flash attention and KV cache tuning to maximize inference throughput on compatible GPUs.

What is the best way to scale large models across multiple GPUs for inference?▼

The best way to scale large models across multiple GPUs for inference is using TensorRT-LLM's multi-GPU and multi-node scaling capabilities. This allows very large models to be distributed efficiently across A100 or H100 clusters.

Does TensorRT-LLM provide an OpenAI-compatible API for serving?▼

Yes, TensorRT-LLM provides an OpenAI-compatible API for serving. It includes production-ready server configuration with Prometheus metrics to monitor performance and meet end-to-end deployment requirements.

Do I need specific NVIDIA GPUs to run TensorRT-LLM for production inference?▼

You need NVIDIA GPUs like A100 or H100 for optimal TensorRT-LLM production inference. The framework is specifically designed to maximize throughput and low latency on these hardware architectures for enterprise deployments.

Related Skills

Explore More Software Engineering Skills→
JuanMS20JuanMS20

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
CyapstayeCyapstaye
9

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints and Prometheus metrics.

Community
Advanced
cloudlinesscloudliness

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
JamesFincherJamesFincher

tensorrt-llm

Optimizes LLM inference serving on NVIDIA GPUs using TensorRT-LLM with quantization and parallelism options.

Community
Advanced