skills.rest
AlexiosBluffMara
AlexiosBluffMaraCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Accelerate LLM inference with TensorRT on NVIDIA GPUs.

2|Updated Apr 25, 2026
Search Tags:#optimization#llm#latency#gpu#throughput#inference#tensorrt
One-click install
npx skills add https://github.com/AlexiosBluffMara/mercury --skill tensorrt-llm-alexiosbluffmara
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/AlexiosBluffMara/mercury/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/AlexiosBluffMara/mercury --skill tensorrt-llm-alexiosbluffmara

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference by leveraging NVIDIA TensorRT to maximize throughput and minimize latency for production deployments.

Core Features & Use Cases

  • TensorRT-accelerated inference with FP8/INT4 quantization for higher throughput and lower memory.
  • In-flight batching and multi-GPU scaling to serve large models efficiently.
  • Production-grade deployment on NVIDIA GPUs (A100/H100) with robust serving and monitoring.

Quick Start

Install the CUDA-enabled TensorRT runtime and start the trtllm-serve server with your model to begin accelerated, production-grade inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference latency on NVIDIA GPUs?▼

You optimize LLM inference latency on NVIDIA GPUs by leveraging TensorRT to maximize throughput, utilizing in-flight batching, and applying FP8 or INT4 quantization for production deployments.

What is in-flight batching for LLM inference?▼

In-flight batching for LLM inference is a TensorRT technique that dynamically processes new requests during generation. It maximizes GPU throughput and minimizes latency for production serving.

Do I need CUDA and TensorRT to serve large language models in production?▼

Yes, serving large language models in production requires CUDA, TensorRT, and compatible Python dependencies. This setup ensures robust, safe serving on NVIDIA GPUs like A100 or H100.

Can I use multi-GPU scaling across single or multi-node clusters for LLM inference?▼

Yes, multi-GPU scaling across single or multi-node clusters is supported for LLM inference. This allows you to serve large models efficiently on NVIDIA A100 and H100 GPUs.

What's the best way to start accelerated inference with TensorRT?▼

The best way to start accelerated inference is to install the CUDA-enabled TensorRT runtime and start the trtllm-serve server with your model for robust, production-grade serving.

Related Skills

Explore More Software Engineering Skills→
m0atm0at
1

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
RheasilviaRheasilvia

tensorrt-llm

Optimize large language model inference on NVIDIA GPUs with quantization and dynamic batching.

Community
Advanced
cloudlinesscloudliness

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
BermudaLocalsBermudaLocals
1

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs using quantization and multi-GPU scaling.

Community
Advanced