skills.rest
zerwiz
zerwizCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs via TensorRT-LLM configuration and deployment.

1|Updated Apr 10, 2026
Search Tags:#quantization#model-serving#llm-inference#latency-optimization#tensorrt-llm#multi-gpu#nvidia-gpus
One-click install
npx skills add https://github.com/zerwiz/wayofpi --skill tensorrt-llm-zerwiz
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/zerwiz/wayofpi/tree/main/.hermes/hermes-agent/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/zerwiz/wayofpi --skill tensorrt-llm-zerwiz

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps teams turn slow, memory-heavy LLM inference into fast, production-ready serving on NVIDIA GPUs, reducing latency and improving throughput.

Core Features & Use Cases

  • Inference optimization: Tune quantization, batching, KV cache usage, and CUDA graphs for better speed and memory efficiency.
  • Multi-GPU scaling: Configure tensor, pipeline, and expert parallelism for larger models and distributed deployments.
  • Production serving: Run OpenAI-compatible endpoints with health checks, metrics, autoscaling, and authentication for real-world workloads.
  • Use case: Deploy a 70B model on H100s with FP8 and in-flight batching for high-traffic chat or agent workloads.

Quick Start

Use the tensorrt-llm skill to plan, configure, and deploy an optimized NVIDIA GPU serving setup for your target model and traffic pattern.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference latency on NVIDIA GPUs?▼

You can optimize LLM inference on NVIDIA GPUs by tuning quantization, in-flight batching, and KV cache usage. This reduces latency and improves throughput for production model serving.

Can I use tensor parallelism and pipeline parallelism for multi-GPU LLM serving?▼

Yes, multi-GPU LLM serving supports tensor, pipeline, and expert parallelism. Configuring these parallelism strategies allows you to scale larger models across distributed deployments efficiently.

What is the best way to deploy a 70B model on H100 GPUs for high-traffic workloads?▼

Deploying a 70B model on H100s for high-traffic workloads involves applying FP8 quantization and in-flight batching. This configuration maximizes throughput and minimizes latency for chat or agent applications.

Does TensorRT-LLM support OpenAI-compatible endpoints with health checks and autoscaling?▼

Yes, production LLM serving includes OpenAI-compatible endpoints equipped with health checks, metrics, and autoscaling. This ensures real-world workloads are handled reliably with built-in authentication.

How does KV cache optimization improve LLM throughput and memory efficiency?▼

KV cache optimization improves LLM throughput by managing memory allocation during token generation. Combined with CUDA graphs and batching, it significantly enhances speed and memory efficiency for inference.

Do I need specific NVIDIA GPU hardware to run TensorRT-LLM for model deployment?▼

Yes, running TensorRT-LLM requires TensorRT-LLM-compatible NVIDIA GPU hardware. Proper hardware ensures you can effectively configure parallelism, caching, and monitoring for optimized model deployment.

Related Skills

Explore More Software Engineering Skills→
JuanMS20JuanMS20

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
hung-phanhung-phan
1

ray-serve

Deploy ML models to Ray Serve with autoscaling and request batching.

Community
Advanced
Supporter09Supporter09

tensorrt-llm

Configure TensorRT-LLM inference with quantization, batching, and multi-GPU serving.

Community
Advanced
brittaniebuffiecsubrittaniebuffiecsu
1

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and quantization using vLLM.

Community
Advanced