skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT/CUDA acceleration.

2|Updated Apr 12, 2026
Search Tags:#quantization#fp8#nvidia#model-serving#llm-inference#tensorrt-llm#multi-gpu
One-click install
npx skills add https://github.com/Clay-HHK/claude-config --skill tensorrt-llm-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Clay-HHK/claude-config/tree/main/skills/AI-research-SKILLs/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/Clay-HHK/claude-config --skill tensorrt-llm-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill eliminates slow, memory-constrained LLM inference by providing a production-ready, NVIDIA-accelerated pipeline that maximizes throughput and minimizes latency when serving large models.

Core Features & Use Cases

  • Performance optimizations: In-flight batching, CUDA graphs, flash attention, and paged KV cache to boost tokens/sec and stabilise P99 latency.
  • Quantization & memory reduction: FP8, INT4, and AWQ/GPTQ workflows to fit larger models and reduce GPU memory usage.
  • Parallelism & scaling: Tensor, pipeline, and expert parallelism patterns for single-node multi-GPU and multi-node deployments.
  • Serving & observability: OpenAI-compatible trtllm-serve, Prometheus metrics, health endpoints, and Kubernetes deployment examples.
  • Use Case: Serve Meta-Llama-3-8B on H100 with FP8 for low-latency chat applications or scale Llama-3-70B across 4 A100s for high-throughput batch inference.

Quick Start

Start trtllm-serve for meta-llama/Meta-Llama-3-8B using fp8 and tp_size=4 to deploy a high-throughput, low-latency inference endpoint.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I reduce LLM inference latency on NVIDIA H100 GPUs?▼

Reduce LLM inference latency on NVIDIA H100 GPUs by applying FP8 quantization, in-flight batching, and CUDA graphs. This pipeline maximizes tokens per second and stabilizes P99 latency for production serving.

What is the best way to serve a 70B model across multiple A100 GPUs?▼

The best way to serve a 70B model across multiple A100 GPUs is using tensor and pipeline parallelism. This approach shards the model across devices to maximize throughput and overcome single-GPU memory limits.

Can I use FP8 quantization with TensorRT-LLM for production model serving?▼

Yes, you can use FP8 quantization with TensorRT-LLM for production model serving. It reduces GPU memory usage and increases throughput on supported hardware like H100 and A100.

Does TensorRT-LLM support OpenAI-compatible API endpoints?▼

Yes, TensorRT-LLM supports OpenAI-compatible API endpoints through trtllm-serve. This allows seamless integration with existing client applications expecting standard OpenAI request formats.

How does in-flight batching improve LLM inference throughput?▼

In-flight batching improves LLM inference throughput by dynamically inserting new requests into ongoing batches. This maximizes GPU utilization and significantly increases tokens per second without waiting for batch completion.

Do I need Kubernetes to deploy models with multi-GPU tensor parallelism?▼

No, you do not strictly need Kubernetes to deploy models with multi-GPU tensor parallelism, as it supports single-node multi-GPU deployments. However, Kubernetes provides deployment examples for scalable observability and management.

Related Skills

Explore More Software Engineering Skills→
ever-oliever-oli
3

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and CUDA graphs.

Community
Advanced
thisismynewfmail-uithisismynewfmail-ui

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT-LLM on A100/H100 GPUs.

Community
Advanced
Open330Open330
1

deploying-triton

Deploy and manage NVIDIA Triton Inference Server containers with health checks.

Official
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference server using vLLM with tensor parallelism and quantization.

Community
Intermediate