skills.rest
ldzhhxx
ldzhhxxCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for production GPUs.

1|1|Updated May 9, 2026
Search Tags:#LLM#production deployment#low latency#TensorRT#inference optimization#high throughput
One-click install
npx skills add https://github.com/ldzhhxx/Hermes_offline_v2 --skill tensorrt-llm-ldzhhxx
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/ldzhhxx/Hermes_offline_v2/tree/main/hermes-agent/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/ldzhhxx/Hermes_offline_v2 --skill tensorrt-llm-ldzhhxx

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides a solution for optimizing LLM inference using NVIDIA TensorRT, offering maximum throughput and lowest latency, suitable for production deployment on NVIDIA GPUs (A100/H100).

Core Features & Use Cases

  • High Throughput and Low Latency: Achieve 10-100x faster inference than PyTorch on NVIDIA GPUs.
  • Quantization: Support for quantization (FP8/INT4) for serving models with reduced size and improved inference speed.
  • In-Flight Batching: Dynamic batching during generation for increased throughput.
  • Multi-GPU Scaling: Scale inference across multiple GPUs or nodes.
  • Use Case: Ideal for scenarios where high performance is required, such as serving models in production environments.

Quick Start

Use the tensorrt-llm skill to optimize inference for a LLM model.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for low latency and high throughput on NVIDIA GPUs?▼

Optimize LLM inference by using NVIDIA TensorRT to build highly optimized engines, achieving 10-100x faster inference than standard PyTorch while significantly reducing latency for production deployment.

What is in-flight batching and how does it improve LLM inference performance?▼

In-flight batching is a dynamic batching mechanism applied during token generation that increases overall inference throughput by continuously scheduling new requests as others complete.

Can I use quantization to reduce LLM size and improve inference speed?▼

Yes, you can apply quantization using FP8 or INT4 formats to reduce model size and improve inference speed, making it more efficient for serving models in production environments.

How do I scale LLM inference across multiple GPUs or nodes?▼

Scale LLM inference across multiple GPUs or nodes by utilizing multi-GPU scaling capabilities, which allows you to distribute the workload and serve models that require high performance.

Do I need specific hardware or frameworks to deploy LLMs with TensorRT?▼

Yes, production deployment requires NVIDIA GPUs such as A100 or H100, along with CUDA, TensorRT, and PyTorch dependencies to properly execute the optimized inference engines.

When should I choose TensorRT over PyTorch for serving LLM models?▼

Choose TensorRT over PyTorch when you require maximum throughput and lowest latency for serving models in production, as it delivers 10-100x faster inference on NVIDIA GPUs.

Related Skills

Explore More Software Engineering Skills→
brittaniebuffiecsubrittaniebuffiecsu
1

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT with FP8/INT4 quantization and multi-GPU scaling.

Community
Advanced
aayushsoamaayushsoam
1

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
loanntcloanntc

machine-learning-strategy

Generates trading direction signals from OHLCV time series using scikit-learn models with walk-forward training.

Community
Intermediate