skills.rest
JKhyro
JKhyroCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference and serving on NVIDIA GPUs with TensorRT-LLM.

Updated Apr 2, 2026
Search Tags:#quantization#low latency#nvidia#llm inference#tensorrt-llm#gpu serving#multi-gpu
One-click install
npx skills add https://github.com/JKhyro/HERMES-AGENT --skill tensorrt-llm-jkhyro
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/JKhyro/HERMES-AGENT/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/JKhyro/HERMES-AGENT --skill tensorrt-llm-jkhyro

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM removes the bottlenecks of slow, memory-hungry LLM inference by turning NVIDIA GPUs into high-throughput, low-latency serving engines for production workloads.

Core Features & Use Cases

  • Inference acceleration: Boosts generation speed with TensorRT optimizations, CUDA graphs, in-flight batching, and paged KV cache management.
  • Production serving: Deploys OpenAI-compatible endpoints with health checks, Prometheus metrics, load balancing, and autoscaling support.
  • Scaling and compression: Supports FP8 and INT4 quantization, tensor and pipeline parallelism, speculative decoding, and multi-GPU or multi-node setups.
  • Use cases: Ideal for serving Llama, Qwen, Mixtral, DeepSeek, and other large models on A100, H100, and similar NVIDIA hardware with strict latency or throughput targets.

Quick Start

Ask the skill to deploy your chosen model with the best TensorRT-LLM configuration for your NVIDIA GPU budget, latency target, and throughput goal.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference on NVIDIA GPUs for low latency?▼

Optimize LLM inference on NVIDIA GPUs by applying TensorRT-LLM features like CUDA graphs, in-flight batching, and paged KV cache. This approach maximizes throughput and minimizes latency for production serving across common transformer families.

How do I serve Llama or Mixtral models with an OpenAI-compatible endpoint?▼

Serve Llama, Mixtral, or other large models using an OpenAI-compatible endpoint configured via TensorRT-LLM. Deployment includes health checks, Prometheus metrics, load balancing, and autoscaling support for robust production serving.

Does TensorRT-LLM support FP8 and INT4 quantization for reducing memory usage?▼

TensorRT-LLM supports FP8 and INT4 quantization for model compression and scaling. These techniques reduce memory consumption and accelerate quantized model execution on compatible NVIDIA hardware.

What's the best way to scale LLM serving across multi-GPU or multi-node setups?▼

Scale LLM serving across multi-GPU or multi-node setups using tensor and pipeline parallelism. TensorRT-LLM coordinates these parallelism strategies alongside speculative decoding to meet strict throughput targets.

Can I use TensorRT-LLM with PyTorch for production deployment?▼

TensorRT-LLM integrates with PyTorch dependencies to deploy optimized models on NVIDIA GPUs. It targets production deployment scenarios requiring high throughput and minimum latency for large language models.

Why does LLM inference run slowly on A100 or H100 GPUs without optimization?▼

LLM inference runs slowly without optimization due to memory bottlenecks and unoptimized execution paths. TensorRT-LLM resolves this by turning NVIDIA GPUs into high-throughput, low-latency serving engines using paged KV cache and in-flight batching.

Related Skills

Explore More Software Engineering Skills→
brittaniebuffiecsubrittaniebuffiecsu
1

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT with FP8/INT4 quantization and multi-GPU scaling.

Community
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
cxnaivecxnaive

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM.

Community
Advanced
BermudaLocalsBermudaLocals
1

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs using quantization and multi-GPU scaling.

Community
Advanced