skills.rest
dawsonblock
dawsonblockCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM on multi-GPU servers.

Updated Apr 26, 2026
Search Tags:#llm#quantization#inference#serving#fp8#tensorrt#multi-gpu
One-click install
npx skills add https://github.com/dawsonblock/HERMY --skill tensorrt-llm-dawsonblock
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/dawsonblock/HERMY/tree/main/hermes-agent-2026.4.23/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/dawsonblock/HERMY --skill tensorrt-llm-dawsonblock

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This skill accelerates large language model inference by leveraging NVIDIA TensorRT-LLM, delivering higher throughput and lower latency for production-grade deployments on GPU hardware.

Core Features & Use Cases

  • High-throughput inference: Optimizes LLMs on NVIDIA GPUs with FP8/INT4 quantization and in-flight batching.
  • Production-grade serving: Supports multi-GPU scaling, CUDA graphs, and OpenAI-compatible APIs for robust deployments.
  • Use case: Deploy a 70B Llama model across 4 A100 GPUs to serve real-time chat workloads with latency under 20 ms.

Quick Start

Install the tensorrt_llm package and launch the server with a target model to begin serving at scale.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I speed up LLM inference for production serving on multi-GPU servers?▼

Speed up LLM inference using NVIDIA TensorRT-LLM to achieve high throughput and low latency on multi-GPU servers. It optimizes models with FP8 quantization, in-flight batching, and scalable serving for robust production deployments.

Does TensorRT-LLM support FP8 quantization and in-flight batching for large language models?▼

Yes, TensorRT-LLM supports FP8 quantization and in-flight batching to optimize large language models. These features maximize inference throughput and reduce latency on NVIDIA GPUs for real-time workloads.

Can I deploy a 70B Llama model across 4 A100 GPUs with OpenAI-compatible APIs?▼

You can deploy a 70B Llama model across 4 A100 GPUs using TensorRT-LLM. It supports multi-GPU scaling and OpenAI-compatible APIs to serve real-time chat workloads with latency under 20 ms.

What is the best way to optimize multi-node LLM serving with low latency?▼

The best way to optimize multi-node LLM serving is using TensorRT-LLM with CUDA graphs and in-flight batching. This approach delivers low latency and scales efficiently across multi-node GPU setups.

Do I need torch and tensorrt-llm dependencies to scale LLM inference?▼

Yes, you need both tensorrt-llm and torch dependencies to scale LLM inference. These packages provide the necessary environment to build optimized engines and execute high-throughput model serving.

How does in-flight batching work for high-throughput LLM inference?▼

In-flight batching works in TensorRT-LLM by dynamically routing requests through CUDA graphs to maximize GPU utilization. This mechanism significantly increases high-throughput LLM inference and reduces latency.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
markwang2658markwang2658
31

tensorrt-llm

Optimize large-language-model inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
kaminocorpkaminocorp
228

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Official
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced