skills.rest
founderphantom
founderphantomCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and FP8/INT4 quantization.

Updated Apr 1, 2026
Search Tags:#production#llm#latency#gpu#throughput#inference#tensorrt
One-click install
npx skills add https://github.com/founderphantom/zola-agent --skill tensorrt-llm-founderphantom
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/founderphantom/zola-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/founderphantom/zola-agent --skill tensorrt-llm-founderphantom

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates large language model inference on NVIDIA GPUs by optimizing runtime, reducing latency, and boosting throughput for production deployments.

Core Features & Use Cases

  • TensorRT-based optimization with FP8/INT4 quantization, in-flight batching, and efficient KV caching.
  • Multi-GPU and multi-node scaling for large models, enabling high-throughput serving in production.
  • Production-ready deployment with an OpenAI-compatible API, monitoring, and autoscaling.

Quick Start

Run a TensorRT-LLM deployment command to launch a high-throughput LLM service on your NVIDIA GPUs.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for low latency and high throughput on NVIDIA GPUs?▼

LLM inference on NVIDIA GPUs is optimized by leveraging TensorRT to apply FP8 or INT4 quantization, in-flight batching, and efficient KV caching, which collectively reduce latency and boost throughput for production deployments.

Can I scale large language model serving across multiple GPUs and nodes?▼

Large language model serving can be scaled across multiple GPUs and nodes. TensorRT-based optimization supports multi-GPU and multi-node scaling on A100 and H100 clusters to enable high-throughput model serving in production.

Does TensorRT-LLM support production features like OpenAI-compatible APIs and monitoring?▼

TensorRT-LLM supports production features including an OpenAI-compatible API, monitoring, and autoscaling. It ensures compatibility with production workflows, CUDA graphs, and reliable live deployments.

What quantization methods are available for accelerating large language models on H100 GPUs?▼

FP8 and INT4 quantization methods are available for accelerating large language models on H100 GPUs. These techniques reduce memory bandwidth usage and increase throughput during TensorRT runtime optimization.

How does in-flight batching improve LLM throughput in production deployments?▼

In-flight batching improves LLM throughput in production by dynamically scheduling inference requests while previous requests are still generating tokens. This mechanism maximizes GPU utilization and reduces overall latency.

Related Skills

Explore More Software Engineering Skills→
brittaniebuffiecsubrittaniebuffiecsu
1

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT with FP8/INT4 quantization and multi-GPU scaling.

Community
Advanced
hung-phanhung-phan
1

ray-serve

Deploy ML models to Ray Serve with autoscaling and request batching.

Community
Advanced
InverterNetworkInverterNetwork

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with quantization and parallelization.

Official
Advanced
arsityarsity

tensorrt-llm

Optimize LLM inference throughput and latency on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced