skills.rest
KarlinskyS
KarlinskySCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs with quantization and multi-GPU scaling.

Updated Apr 10, 2026
Search Tags:#llm#inference#fp8#nvidia#tensorrt#int4#multi-gpu
One-click install
npx skills add https://github.com/KarlinskyS/hermesSkills --skill tensorrt-llm-karlinskys
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/KarlinskyS/hermesSkills/tree/main/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/KarlinskyS/hermesSkills --skill tensorrt-llm-karlinskys

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deploying and serving large language models on NVIDIA GPUs for production use requires extreme throughput, low latency, efficient memory usage, and careful handling of quantization and parallelism; achieving this with PyTorch alone is often too slow or memory-intensive. TensorRT-LLM compiles and optimizes models for NVIDIA hardware, enabling dramatic speedups (10–100× vs PyTorch), reduced memory footprint with FP8/INT4, and reliable multi-GPU scaling for production inference.

Core Features & Use Cases

  • High-performance inference: Model compilation, CUDA graphs, flash attention, and in-flight batching to maximize tokens/sec and stabilize P99 latency.
  • Quantized serving: FP8, INT4 and AWQ/GPTQ workflows to reduce memory and increase throughput for H100/A100 deployments.
  • Multi-GPU & multi-node scaling: Tensor parallelism, pipeline parallelism, expert parallelism, NVLink/InfiniBand guidance and Kubernetes/Ray deployment patterns.
  • Production readiness: OpenAI-compatible trtllm-serve, Prometheus metrics, health probes, autoscaling examples, and best-practice security and cost-optimization tips.
  • Use case example: Serve a Meta Llama 3-70B model across 4 A100/H100 GPUs with FP8 quantization for high-throughput chat completions and low-latency streaming.

Quick Start

Start a TensorRT-LLM server hosting meta-llama/Meta-Llama-3-8B with FP8 and tensor parallelism across 4 GPUs for high-throughput, low-latency inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference throughput and minimize latency on NVIDIA GPUs?▼

Optimizing LLM inference on NVIDIA GPUs involves compiling models with TensorRT-LLM to leverage CUDA graphs, flash attention, and in-flight batching. This approach maximizes tokens per second and stabilizes P99 latency for production deployments.

Does TensorRT-LLM support FP8 and INT4 quantization for serving large models on A100 or H100?▼

Yes, FP8 and INT4 quantization with AWQ and GPTQ workflows are fully supported. These quantization techniques reduce the memory footprint and increase throughput for serving large models on A100 and H100 hardware.

What is the best way to scale multi-GPU LLM inference for production?▼

Scaling multi-GPU LLM inference is best handled using tensor parallelism, pipeline parallelism, and expert parallelism. TensorRT-LLM provides guidance for NVLink and InfiniBand setups alongside Kubernetes and Ray deployment patterns.

Can I use an OpenAI-compatible API for serving quantized LLM models?▼

Yes, you can use the OpenAI-compatible trtllm-serve feature for serving quantized LLM models. This includes Prometheus metric integration, health probes, and autoscaling examples to ensure production readiness and reliable deployments.

How much faster is TensorRT-LLM compared to PyTorch for LLM inference?▼

TensorRT-LLM achieves dramatic speedups of 10 to 100 times compared to PyTorch for LLM inference. It compiles and optimizes models specifically for NVIDIA hardware, overcoming the slow and memory-intensive limitations of PyTorch.

Related Skills

Explore More Software Engineering Skills→
math-incmath-inc
1.2k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Official
Advanced
hung-phanhung-phan
1

ray-serve

Deploy ML models to Ray Serve with autoscaling and request batching.

Community
Advanced
aws-samplesaws-samples
18

vllm-serving-setup

Automate vLLM inference serving deployment on EKS Kubernetes clusters.

Official
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced