skills.rest
zulumonkeymetallic
zulumonkeymetallicCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference with TensorRT on NVIDIA GPUs.

Updated Aug 23, 2026
Search Tags:#quantization#production-deployment#gpu-acceleration#inference-optimization#tensor-parallelism#tensorrt-llm
One-click install
npx skills add https://github.com/zulumonkeymetallic/bob --skill tensorrt-llm-zulumonkeymetallic
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/zulumonkeymetallic/bob/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/zulumonkeymetallic/bob --skill tensorrt-llm-zulumonkeymetallic

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

NVIDIA TensorRT-LLM addresses the need for production-grade LLM inference that delivers high throughput and low latency on NVIDIA GPUs. It optimizes models with TensorRT, supports quantization (FP8/INT4/FP4), in-flight batching, and multi-GPU scaling to serve large models efficiently.

Core Features & Use Cases

  • Tensor parallelism across GPUs for large models
  • Quantization support (FP8/INT4/FP4) for memory and speed
  • In-flight batching and CUDA graphs for higher utilization
  • Multi-node deployment for scale-out inference
  • Use Case: Deploy a flagship LLM on A100/H100 clusters to serve chat or generation workloads with low latency

Quick Start

Install TensorRT-LLM and run a basic inference to verify deployment.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I speed up LLM inference on NVIDIA A100 or H100 GPUs?▼

To speed up LLM inference on NVIDIA A100 or H100 GPUs, you can leverage TensorRT for high throughput and low latency. It applies in-flight batching and CUDA graphs to maximize GPU utilization during production deployment.

What is the best way to reduce LLM memory usage without losing inference speed?▼

Reducing LLM memory usage while maintaining speed requires quantization. You can apply FP8, INT4, or FP4 quantization within TensorRT to decrease memory footprints and accelerate generation workloads on supported NVIDIA hardware.

Can I use tensor parallelism to deploy large LLMs across multiple GPUs?▼

Yes, you can use tensor parallelism to deploy large LLMs across multiple GPUs. This approach distributes model execution across NVIDIA hardware and supports multi-node deployment for scale-out inference.

Does TensorRT-LLM require specific frameworks like PyTorch to run?▼

TensorRT-LLM requires PyTorch and the TensorRT-LLM library as dependencies. You need these frameworks installed in your environment to optimize models and execute high-throughput inference workloads.

When do I need in-flight batching for production LLM serving?▼

You need in-flight batching for production LLM serving when handling continuous chat or generation workloads. This technique dynamically processes requests to improve GPU utilization and reduce latency for concurrent users.

Why is my LLM inference slow during production deployment?▼

LLM inference is often slow without optimized execution graphs and batching. Leveraging TensorRT with CUDA graphs and in-flight batching resolves this by maximizing GPU utilization and minimizing latency bottlenecks.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
math-incmath-inc
1.2k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Official
Advanced
hung-phanhung-phan
1

triton-inference-server

Serve ML models with dynamic batching and concurrent execution via Triton Inference Server.

Community
Advanced
carterwayneskhizeinecarterwayneskhizeine
19

serving-llms-vllm

Deploy an OpenAI-compatible LLM inference server using vLLM with tensor parallelism and quantization.

Community
Intermediate