skills.rest
AlvaroBiano
AlvaroBianoCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Accelerate LLM inference on NVIDIA GPUs with TensorRT-LLM optimizations.

Updated May 2, 2026
Search Tags:#quantization#gpu-acceleration#high-throughput#llm-inference#tensorrt-llm#multi-gpu
One-click install
npx skills add https://github.com/AlvaroBiano/hermes-agent --skill tensorrt-llm-alvarobiano
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/AlvaroBiano/hermes-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/AlvaroBiano/hermes-agent --skill tensorrt-llm-alvarobiano

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Accelerate LLM inference performance on NVIDIA GPUs.

Core Features & Use Cases

  • TensorRT-LLM optimizations: In-flight batching, quantization (FP8/INT4), and tensor parallelism for large models.
  • Production serving: Ready-to-use serving with a fast OpenAI-compatible API via trtllm-serve.
  • Use cases: Deploy massive models (70B–405B) at multi-GPU scale with low latency and cost efficiency.

Quick Start

Install the TensorRT-LLM stack and run the server with a target model to begin production inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production workloads?▼

Accelerate LLM inference on NVIDIA GPUs by applying TensorRT-LLM optimizations like in-flight batching, FP8/INT4 quantization, and tensor parallelism to achieve high throughput and low latency. This Skill guides deployment for production serving via an OpenAI-compatible API.

What is in-flight batching and how does it improve high-throughput LLM inference?▼

In-flight batching is a TensorRT-LLM optimization that dynamically schedules requests during generation to maximize GPU utilization. It improves high-throughput LLM inference by reducing latency and increasing overall token generation rates for production workloads.

Can I deploy massive 70B to 405B parameter models with multi-GPU scaling?▼

Yes, you can deploy massive 70B to 405B parameter models using multi-GPU scaling and tensor parallelism. This approach distributes the computational load across multiple NVIDIA GPUs to maintain low latency and cost efficiency for large-scale production serving.

How do I use FP8 or INT4 quantization to reduce LLM deployment costs?▼

Apply FP8 or INT4 quantization through TensorRT-LLM to reduce LLM deployment costs by lowering memory bandwidth requirements and accelerating inference. This allows large models to run efficiently on NVIDIA GPUs without significant accuracy loss.

Does this provide an OpenAI-compatible API for serving LLMs?▼

Yes, it provides ready-to-use production serving with a fast OpenAI-compatible API via trtllm-serve. You can install the TensorRT-LLM stack and run the server with a target model to begin production inference immediately.

What is the best way to optimize large model serving for low latency and high throughput?▼

The best way to optimize large model serving is combining TensorRT-LLM in-flight batching, FP8/INT4 quantization, and multi-GPU tensor parallelism. This configuration maximizes throughput and minimizes latency for demanding production inference workloads on NVIDIA GPUs.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
gagan114662gagan114662

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
cloudlinesscloudliness

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced