skills.rest
carterwayneskhizeine
carterwayneskhizeineCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT-LLM for throughput and latency.

19|4|Updated Apr 22, 2026
Search Tags:#tensorrt-llm#nvidia gpu#inference serving#fp8 quantization#multi-gpu scaling#int4 int4_awq#in-flight batching
One-click install
npx skills add https://github.com/carterwayneskhizeine/hermes-agent-windows-R --skill tensorrt-llm-carterwayneskhizeine
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/carterwayneskhizeine/hermes-agent-windows-R/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/carterwayneskhizeine/hermes-agent-windows-R --skill tensorrt-llm-carterwayneskhizeine

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM solves the performance bottleneck of slow LLM inference by optimizing model execution to achieve higher throughput and lower latency on NVIDIA hardware.

Core Features & Use Cases

  • In-flight batching and efficient KV cache: Improves utilization and supports higher request concurrency during generation (useful for chat workloads and mixed prompt lengths).
  • Low-precision quantization (FP8/INT4/FP4): Reduces memory footprint while accelerating inference for production-scale serving.
  • Production-ready serving: Runs an OpenAI-compatible server endpoint with optimized scheduling, metrics, and health checks (useful for deploying Llama/Qwen/DeepSeek-class models behind an API).

Quick Start

Run the TensorRT-LLM server for your model with tensor parallelism and FP8 enabled for maximum throughput on NVIDIA GPUs by executing: start trtllm-serve <model> --tp_size <num_gpus> --dtype fp8 --max_batch_size <batch>.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I speed up LLM inference on NVIDIA GPUs for production serving?▼

Speed up LLM inference on NVIDIA GPUs by applying TensorRT-LLM optimizations like in-flight batching and FP8 or INT4 quantization to significantly increase throughput and reduce latency during model execution.

How does in-flight batching improve LLM request concurrency?▼

In-flight batching improves LLM request concurrency by dynamically scheduling generation tokens with efficient KV cache management, which increases GPU utilization and supports mixed prompt lengths in chat workloads.

Can I deploy an OpenAI-compatible API endpoint for Llama models using TensorRT-LLM?▼

Yes, you can deploy an OpenAI-compatible API endpoint for Llama models using the TensorRT-LLM server, which provides optimized scheduling, health checks, and metrics for production-scale serving.

Do I need a specific NVIDIA software stack to use FP8 quantization with TensorRT-LLM?▼

Yes, FP8 quantization requires compatible NVIDIA GPU software stacks to execute the optimized low-precision models and achieve the reduced memory footprint and accelerated inference speeds.

What is the best way to start serving a multi-GPU LLM with tensor parallelism?▼

The best way to start serving a multi-GPU LLM is executing the trtllm-serve command with tensor parallelism, specifying your model, GPU count, data type like FP8, and maximum batch size.

Why use INT4 or FP8 quantization for LLM inference?▼

Use INT4 or FP8 quantization for LLM inference to reduce the model memory footprint and accelerate generation speeds during production-scale serving without heavily compromising execution efficiency.

Related Skills

Explore More Software Engineering Skills→
DaddyElonMusk69DaddyElonMusk69

tensorrt-llm

Accelerate LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Intermediate
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
automatedigitalautomatedigital
1

serving-llms-vllm

Serve large language models with vLLM's optimized inference engine.

Official
Advanced
ricablericable

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and parallelism.

Community
Advanced