skills.rest
BermudaLocals
BermudaLocalsCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs using quantization and multi-GPU scaling.

1|1|Updated Apr 26, 2026
Search Tags:#deployment#production#optimization#llm#gpu#inference#tensorrt
One-click install
npx skills add https://github.com/BermudaLocals/hermes-agent-lite --skill tensorrt-llm-bermudalocals
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/BermudaLocals/hermes-agent-lite/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/BermudaLocals/hermes-agent-lite --skill tensorrt-llm-bermudalocals

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM provides production-grade optimization for LLM inference on NVIDIA GPUs, reducing latency and increasing throughput through TensorRT, quantization, and advanced serving features.

Core Features & Use Cases

  • In-flight batching and optimized KV cache for scalable, low-latency inference across GPUs.
  • FP8 and INT4 quantization to minimize memory footprint while preserving accuracy.
  • Multi-GPU and multi-node deployment guidance using tensor parallelism, pipeline parallelism, and expert parallelism for large models.

Quick Start

Start the TensorRT-LLM server with an FP8-quantized model on a multi-GPU setup.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for lower latency on NVIDIA GPUs?▼

Optimize LLM inference on NVIDIA GPUs by leveraging TensorRT to maximize throughput and minimize latency using in-flight batching, optimized KV cache, and FP8 or INT4 quantization in production environments.

What is the best way to deploy large models across multiple GPUs?▼

Deploy large models across multiple GPUs or nodes using tensor parallelism, pipeline parallelism, and expert parallelism to scale LLM inference efficiently and handle increased production workloads.

Can I use FP8 and INT4 quantization to reduce LLM memory footprint?▼

Yes, you can apply FP8 and INT4 quantization to minimize the LLM memory footprint while preserving accuracy, enabling more efficient inference on NVIDIA A100 and H100 GPUs.

How does in-flight batching improve LLM throughput?▼

In-flight batching improves LLM throughput by dynamically processing new requests alongside ongoing ones, maximizing GPU utilization and reducing latency for scalable inference.

Do I need A100 or H100 GPUs for production LLM serving with TensorRT?▼

You need NVIDIA GPUs like A100 or H100 for production LLM serving with TensorRT, as these provide the necessary hardware acceleration for quantization, tensor parallelism, and low-latency inference.

What are the limitations of using TensorRT for LLM optimization?▼

TensorRT for LLM optimization requires specific NVIDIA GPU hardware and dependencies like tensorrt-llm and torch, limiting deployment flexibility to compatible GPU environments rather than general-purpose platforms.

Related Skills

Explore More Software Engineering Skills→
math-incmath-inc
1.2k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Official
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
thisismynewfmail-uithisismynewfmail-ui

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT-LLM on A100/H100 GPUs.

Community
Advanced
RheasilviaRheasilvia

serving-llms-vllm

Deploy OpenAI-compatible LLM inference servers using vLLM with PagedAttention and continuous batching.

Community
Advanced