skills.rest
zangjeicy
zangjeicyCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize large language model inference on NVIDIA GPUs with TensorRT-LLM.

Updated Jul 13, 2026
Search Tags:#optimization#llm#gpu#inference#serving#nvidia#tensorrt-llm
One-click install
npx skills add https://github.com/zangjeicy/Hermes --skill tensorrt-llm-zangjeicy
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/zangjeicy/Hermes/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/zangjeicy/Hermes --skill tensorrt-llm-zangjeicy

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This skill addresses the performance bottlenecks of running large language models in production by providing high-throughput, low-latency inference optimization specifically for NVIDIA hardware.

Core Features & Use Cases

  • Inference Optimization: Leverages TensorRT-LLM to achieve up to 100x faster inference compared to standard PyTorch implementations.
  • Advanced Serving: Supports in-flight batching, paged KV cache, and multi-GPU parallelism (TP/PP) for production-grade deployments.
  • Use Case: Deploying a Llama 3-70B model on an H100 cluster to serve real-time requests with FP8 quantization, significantly reducing memory footprint and latency.

Quick Start

Use the tensorrt-llm skill to optimize and serve the meta-llama/Meta-Llama-3-8B model with a batch size of 256.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize large language model inference speed on NVIDIA GPUs?▼

Optimize large language model inference on NVIDIA GPUs by leveraging TensorRT-LLM and CUDA-accelerated kernels to achieve maximum throughput and minimal latency for production-scale deployments.

What is the best way to serve a Llama 3 model with FP8 quantization on an H100 cluster?▼

Serving a Llama 3 model with FP8 quantization on an H100 cluster is best achieved using TensorRT-LLM, which significantly reduces memory footprint and latency while supporting real-time request serving.

Does TensorRT-LLM support multi-node parallelism and dynamic in-flight batching?▼

TensorRT-LLM supports multi-node parallelism, dynamic in-flight batching, and paged KV cache to handle high-performance AI serving environments and production-grade multi-GPU deployments.

How does TensorRT-LLM compare to standard PyTorch for LLM inference performance?▼

Compared to standard PyTorch implementations, TensorRT-LLM provides inference optimization that achieves up to 100x faster performance for large language models on NVIDIA hardware.

Can I use INT4 quantization for large language model serving with TensorRT-LLM?▼

You can use INT4 quantization for large language model serving with TensorRT-LLM to maximize inference throughput and minimize latency on NVIDIA GPU architectures.

What are the limitations of using TensorRT-LLM for inference optimization?▼

Inference optimization with TensorRT-LLM is limited to NVIDIA GPU architectures and requires dependencies like torch, making it unsuitable for non-NVIDIA hardware environments.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
Z43LZ43L

tensorrt-llm

Deploys TensorRT-LLM to optimize LLM inference on NVIDIA GPUs.

Community
Advanced
hung-phanhung-phan
1

triton-inference-server

Serve ML models with dynamic batching and concurrent execution via Triton Inference Server.

Community
Advanced
GregsGreyCodeGregsGreyCode
6

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT.

Community
Advanced