skills.rest
attentiondotnet
attentiondotnetCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated Mar 30, 2026
Search Tags:#llm#quantization#inference#serving#fp8#tensorrt#multigpu
One-click install
npx skills add https://github.com/attentiondotnet/hermes-agent --skill tensorrt-llm-attentiondotnet
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/attentiondotnet/hermes-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/attentiondotnet/hermes-agent --skill tensorrt-llm-attentiondotnet

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimize LLM inference on NVIDIA GPUs using TensorRT-LLM to maximize throughput and minimize latency in production deployments.

Core Features & Use Cases

  • In-flight batching, quantization (FP8/INT4), multi-GPU and multi-node deployment, and CUDA graphs for scalable real-time inference.
  • Production serving with OpenAI-friendly API integration and monitoring guidance.
  • Use case: Deploy a 70B model across 4 A100 GPUs to achieve high throughput in a live chat service.

Quick Start

Start the server with trtllm-serve, specify your model, and choose tensor-parallel and data type settings to deploy production-ready LLM inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference on NVIDIA GPUs for maximum throughput?▼

Optimize LLM inference on NVIDIA GPUs using TensorRT-LLM to maximize throughput and minimize latency. It leverages in-flight batching and CUDA graphs to achieve scalable real-time inference for production deployments.

Can I deploy a 70B model across multiple GPUs for production serving?▼

Yes, you can deploy a 70B model across multiple GPUs for production serving. The Skill supports multi-GPU and multi-node deployments using tensor parallelism to distribute the workload and achieve high throughput.

Does TensorRT-LLM support FP8 and INT4 quantization to reduce latency?▼

Yes, TensorRT-LLM supports FP8 and INT4 quantization to reduce latency. These quantization techniques lower memory bandwidth requirements and accelerate inference speeds on NVIDIA GPUs without significantly degrading model accuracy.

How do I start serving an LLM with an OpenAI-compatible API?▼

Start serving an LLM with an OpenAI-compatible API using the trtllm-serve command. Specify your model, choose tensor-parallel and data type settings, and deploy production-ready inference with integrated monitoring guidance.

What is the best way to handle real-time LLM batching in production?▼

The best way to handle real-time LLM batching in production is using in-flight batching. This technique dynamically manages incoming requests to maximize GPU utilization and ensure low-latency responses across live chat services.

Related Skills

Explore More Software Engineering Skills→
peteromalletpeteromallet
97

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
zerwizzerwiz

serving-llms-vllm

Deploy high-throughput LLM inference servers with vLLM and OpenAI-compatible endpoints.

Community
Advanced
CODE-SAURABHCODE-SAURABH
2

system-design

Design scalable software architectures with explicit scale targets and failure tolerance.

Community
Advanced
BermudaLocalsBermudaLocals
1

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs using quantization and multi-GPU scaling.

Community
Advanced