skills.rest
gqf2008
gqf2008COMMUNITY
Explore all skills by this author →
Website𝕏@gqf25044395GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and multi-GPU scaling.

Updated Apr 19, 2026
Search Tags:#production#latency#gpu#throughput#quantization#inference#tensorrt
One-click install
npx skills add https://github.com/gqf2008/hermez-ai --skill tensorrt-llm-gqf2008
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/gqf2008/hermez-ai/tree/main/skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/gqf2008/hermez-ai --skill tensorrt-llm-gqf2008

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM helps production teams deploy high-throughput, low-latency LLM inference on NVIDIA GPUs by optimizing models and serving workflows.

Core Features & Use Cases

  • Performance optimizations: In-flight batching, KV cache optimizations, Flash Attention, CUDA graphs for reduced kernel launch overhead.
  • Parallelism and scaling: Tensor parallelism, pipeline parallelism, expert parallelism, and multi-node support to scale across GPUs and nodes.
  • Quantization and deployment: FP8/INT4/FP4 quantization for faster inference and smaller memory footprint, with support for multi-model serving and production-grade serving.

Quick Start

Install the tensorrt-llm package and start a deployment on NVIDIA GPUs to observe high throughput and low latency.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput and minimize latency on NVIDIA GPUs?▼

To maximize LLM inference throughput and minimize latency on NVIDIA GPUs, you can apply TensorRT-LLM optimizations like in-flight batching, KV cache management, and Flash Attention to reduce kernel launch overhead and accelerate production serving.

Can I scale LLM inference across multiple GPUs and nodes for production deployments?▼

Yes, you can scale LLM inference across multiple GPUs and nodes using tensor parallelism, pipeline parallelism, expert parallelism, and multi-node orchestration to distribute workloads efficiently on A100 and H100 hardware.

How does quantization with FP8 or INT4 improve LLM inference performance?▼

Quantization with FP8, INT4, or FP4 improves LLM inference performance by reducing memory footprint and increasing computational speed, resulting in faster generation and more efficient GPU resource utilization.

What's the best way to optimize production LLM serving on A100 and H100 hardware?▼

The best way to optimize production LLM serving on A100 and H100 hardware is to use CUDA graphs, Flash Attention, and in-flight batching to reduce kernel launch overhead and maximize multi-GPU serving throughput.

Does TensorRT-LLM work with Torch models for GPU-accelerated inference?▼

Yes, TensorRT-LLM works with Torch models to enable GPU-accelerated inference by optimizing Torch-based workflows with quantization, KV cache optimization, and scalable multi-GPU deployment for high-throughput serving.

When do I need in-flight batching and KV cache optimization for LLM inference?▼

You need in-flight batching and KV cache optimization for LLM inference when production deployments require high throughput and low latency, allowing dynamic request scheduling and efficient memory management across concurrent users.

Related Skills

Explore More Software Engineering Skills→
BermudaLocalsBermudaLocals
1

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs using quantization and multi-GPU scaling.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced