skills.rest
CUexter
CUexterCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Accelerate LLM inference on NVIDIA GPUs with TensorRT-LLM.

2|Updated Mar 26, 2026
Search Tags:#production#llm#gpu#quantization#inference#fp8#tensorrt
One-click install
npx skills add https://github.com/CUexter/hermes-agent --skill tensorrt-llm-cuexter
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/CUexter/hermes-agent/tree/main/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/CUexter/hermes-agent --skill tensorrt-llm-cuexter

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates production-grade LLM inference on NVIDIA GPUs, delivering higher throughput and lower latency than CPU-based or non-accelerated paths.

Core Features & Use Cases

  • In-flight batching, CUDA graphs, and quantization (FP8/INT4) for high throughput and efficiency.
  • Multi-GPU and multi-node deployment through TensorRT-LLM with various parallelism strategies.
  • Production serving scenarios via trtllm-serve, OpenAI-compatible API, and monitoring integrations.
  • Supports quantization options and model compatibility across LLaMA, GPT-like models, and others.

Quick Start

Start the production server with a selected model and appropriate tensor-parallelism and quantization settings.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production?▼

To accelerate LLM inference on NVIDIA GPUs, you can use TensorRT-LLM. It delivers higher throughput and lower latency than non-accelerated paths by utilizing in-flight batching, CUDA graphs, and FP8 or INT4 quantization for production deployments.

What's the best way to scale LLM inference across multiple GPUs?▼

The best way to scale LLM inference across multiple GPUs is using TensorRT-LLM. It supports multi-GPU and multi-node deployment through various parallelism strategies, ensuring high throughput and low latency for demanding production environments.

Does TensorRT-LLM support FP8 and INT4 quantization for large language models?▼

Yes, TensorRT-LLM supports FP8 and INT4 quantization. These options help maximize LLM throughput and efficiency, maintaining compatibility across models like LLaMA, GPT-like architectures, and others during production serving.

Can I serve LLM models with an OpenAI-compatible API using TensorRT?▼

Yes, you can serve LLM models with an OpenAI-compatible API using TensorRT-LLM. The trtllm-serve feature provides production serving scenarios, allowing you to start a server with selected models, tensor-parallelism, and quantization settings alongside monitoring integrations.

When do I need in-flight batching for LLM inference?▼

You need in-flight batching for LLM inference when optimizing for high throughput and low latency in production. TensorRT-LLM uses this technique alongside CUDA graphs to efficiently process incoming requests on NVIDIA GPUs without waiting for entire batches to finish.

Related Skills

Explore More Software Engineering Skills→
MesferAliMesferAli

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT-LLM on NVIDIA GPUs.

Community
Advanced
devMoezdevMoez
1

serving-llms-vllm

Deploy vLLM to serve LLMs through an OpenAI-compatible API.

Community
Intermediate
Chris-Chai-MinjaeChris-Chai-Minjae

tensorrt-llm

Accelerate large language model inference on NVIDIA GPUs with FP8/INT4 quantization.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced