skills.rest
Tommy-yw
Tommy-ywCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

539|39|Updated May 1, 2026
Search Tags:#llm#performance optimization#quantization#nvidia#tensorrt#inference serving
One-click install
npx skills add https://github.com/Tommy-yw/RunbookHermes --skill tensorrt-llm-tommy-yw
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Tommy-yw/RunbookHermes/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/Tommy-yw/RunbookHermes --skill tensorrt-llm-tommy-yw

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes scripts (resource) and references (resource) and assets (resource) components.

What problem does it solve?

This Skill optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency, ideal for production deployment on NVIDIA GPUs and serving models with quantization and in-flight batching.

Core Features & Use Cases

  • High Throughput and Low Latency: Accelerates inference on NVIDIA GPUs by up to 100x faster than PyTorch.
  • Quantization: Supports FP8, INT4, and FP4 quantization for reduced memory usage and faster inference.
  • In-Flight Batching: Dynamically batches requests during generation for higher throughput.
  • Multi-GPU Scaling: Scales across multiple GPUs or nodes for large-scale inference.
  • Use Case: Use this Skill to serve a large language model like Meta-Llama 3-70B in a production environment, achieving real-time latency and high throughput.

Quick Start

Install TensorRT-LLM and serve a Meta-Llama 3-8B model:

pip install tensorrt_llm==1.2.0rc3
trtllm-serve meta-llama/Meta-Llama-3-8B \
  --tp_size 4 \
  --max_batch_size 256 \
  --max_num_tokens 4096

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize large language model inference for low latency on NVIDIA GPUs?▼

Optimize large language model inference on NVIDIA GPUs by converting models to TensorRT, achieving up to 100x faster performance than PyTorch for low-latency production deployment.

What is in-flight batching and how does it improve LLM serving throughput?▼

In-flight batching is a dynamic request scheduling technique that batches tokens during generation. It significantly improves LLM serving throughput by continuously feeding the GPU as requests complete.

Can I use FP8 or INT4 quantization to reduce memory usage during LLM inference?▼

Yes, you can use FP8, INT4, and FP4 quantization to reduce memory usage and accelerate inference speeds when deploying large language models on supported NVIDIA GPUs.

How do I serve a Meta-Llama 3 model across multiple GPUs for high-throughput inference?▼

Serve Meta-Llama models across multiple GPUs by using TensorRT-LLM with tensor parallelism. Specify the tensor parallel size in the serving command to scale large-scale inference.

Do I need specific NVIDIA GPUs like A100 or H100 for TensorRT LLM inference?▼

TensorRT LLM inference targets NVIDIA GPUs like A100 and H100 to deliver maximum throughput and real-time latency for large-scale production deployment scenarios.

What is the best way to scale large-scale LLM inference across multiple nodes?▼

Scale large-scale LLM inference across multiple nodes by leveraging TensorRT-LLM's multi-GPU scaling capabilities, which distribute workloads to maximize throughput for real-time applications.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
hung-phanhung-phan
1

triton-inference-server

Serve ML models with dynamic batching and concurrent execution via Triton Inference Server.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced