skills.rest
travelinman1013
travelinman1013COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated Aug 27, 2026
Search Tags:#gpu-acceleration#inference-optimization#tensorrt-llm#fp8-quantization#production-serving#prometheus-metrics
One-click install
npx skills add https://github.com/travelinman1013/leroys-agent --skill tensorrt-llm-travelinman1013
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/travelinman1013/leroys-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/travelinman1013/leroys-agent --skill tensorrt-llm-travelinman1013

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM enables production-grade, high-throughput LLM inference on NVIDIA GPUs by applying advanced optimizations, quantization, and multi-GPU parallelism.

Core Features & Use Cases

  • Performance optimizations: FP8/INT4 quantization, in-flight batching, CUDA graphs for maximum throughput and minimal latency.
  • Parallelism and scalability: Tensor parallelism, pipeline parallelism, and expert parallelism across single-node and multi-node clusters.
  • Production serving: OpenAI-compatible API, Prometheus metrics, health checks, and autoscaling guidance for deployment at scale.

Quick Start

Start serving a model with trtllm-serve using your preferred model, tensor_parallel_size, and dtype to launch a scalable inference endpoint.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs?▼

Maximize LLM inference throughput on NVIDIA GPUs with TensorRT-LLM by applying FP8 and INT4 quantization, in-flight batching, and CUDA graphs to minimize latency for production-scale workloads.

How does FP8 quantization and KV cache tuning optimize model serving?▼

FP8 quantization and KV cache tuning optimize model serving by reducing memory footprint and accelerating computation, enabling model-agnostic inference with higher throughput on compatible NVIDIA GPUs.

Can I deploy multi-GPU parallelism for LLM serving across a cluster?▼

Yes, you can deploy multi-GPU parallelism across single-node and multi-node clusters using tensor parallelism, pipeline parallelism, and expert parallelism to scale LLM inference for production workloads.

Does TensorRT-LLM provide an OpenAI-compatible API for production deployment?▼

Yes, TensorRT-LLM provides an OpenAI-compatible API alongside Prometheus metrics, health checks, and autoscaling guidance to support real-time serving and deployment at scale.

What NVIDIA driver and CUDA toolkit versions are required for production LLM serving?▼

Production LLM serving requires CUDA-enabled GPUs with compatible CUDA toolkit and driver versions to ensure proper hardware acceleration and framework compatibility.

What is the best way to start serving a model using trtllm-serve?▼

Start serving a model using trtllm-serve by specifying your preferred model, tensor_parallel_size, and dtype to launch a scalable inference endpoint for production workloads.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
jleechanorgjleechanorg

serving-llms-vllm

Serve LLMs via vLLM with OpenAI-compatible endpoints and Prometheus monitoring.

Official
Advanced
tadod12tadod12

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for production GPU serving.

Community
Advanced
projectedanxprojectedanx
1

serving-llms-vllm

Deploy vLLM inference servers with OpenAI-compatible API endpoints and continuous batching.

Community
Advanced