skills.rest
icyzh
icyzhCOMMUNITY
Explore all skills by this author →
Website𝕏@icyzhbeGitHub

tensorrt-llm

Optimize large language model inference on NVIDIA GPUs with quantization and parallelism.

1|Updated Jul 31, 2026
Search Tags:#llm#quantization#inference#nvidia#high-throughput#tensorrt-llm
One-click install
npx skills add https://github.com/icyzh/hermes-web --skill tensorrt-llm-icyzh
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/icyzh/hermes-web/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/icyzh/hermes-web --skill tensorrt-llm-icyzh

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This skill addresses the challenge of deploying large language models in production environments where high throughput, low latency, and efficient GPU memory utilization are critical requirements.

Core Features & Use Cases

  • Inference Optimization: Leverages advanced techniques like FP8/INT4 quantization, in-flight batching, and Paged KV cache to maximize performance.
  • Multi-GPU Scaling: Supports tensor, pipeline, and expert parallelism to distribute massive models across multiple GPUs and nodes.
  • Use Case: Deploying a Llama 3-70B model on an H100 cluster to achieve 24,000 tokens per second for real-time, high-concurrency chat applications.

Quick Start

Use the tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with a maximum batch size of 256 on your local GPU.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for high-throughput on NVIDIA GPUs?▼

You can optimize LLM inference on NVIDIA GPUs by applying FP8 and INT4 quantization, in-flight batching, and Paged KV cache to maximize throughput and minimize latency. This skill leverages these techniques for production-grade deployment.

What is the best way to scale large language models across multiple GPUs?▼

Scaling large language models across multiple GPUs is achieved through tensor parallelism, pipeline parallelism, and expert parallelism. These techniques distribute massive models across multiple GPUs and nodes for efficient distributed computing.

Can I use tensor parallelism and FP8 quantization together for LLM deployment?▼

Yes, you can use tensor parallelism and FP8 quantization together for LLM deployment. This skill facilitates combining multi-GPU scaling with advanced quantization techniques to satisfy memory management and scalable serving requirements.

How do I deploy a Llama 3 model on an H100 cluster for real-time chat?▼

To deploy a Llama 3 model on an H100 cluster for real-time chat, you utilize high-throughput inference optimization and multi-GPU scaling. A Llama 3-70B model can achieve 24,000 tokens per second for high-concurrency applications.

Does high-throughput LLM inference support pipeline parallelism for distributed computing?▼

Yes, high-throughput LLM inference supports pipeline parallelism for distributed computing environments. This approach, alongside tensor and expert parallelism, enables efficient memory management and scalable model serving across multiple nodes.

Why does LLM inference latency increase under high concurrency?▼

LLM inference latency often increases under high concurrency due to memory bottlenecks and inefficient batching. Implementing in-flight batching, Paged KV cache, and INT4 quantization resolves these issues by optimizing memory utilization and throughput.

Related Skills

Explore More Software Engineering Skills→
zhuangbiaoweizhuangbiaowei
2

training-llms-megatron

Train large-scale language models using NVIDIA Megatron-Core with advanced parallelism.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
heysuhasheysuhas
2

serving-llms-vllm

Deploy LLM inference servers with the vLLM engine and OpenAI-compatible APIs.

Community
Advanced
davpatel605-beepdavpatel605-beep

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced