skills.rest
heysuhas
heysuhasCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM on GPU hardware.

2|1|Updated Jul 14, 2026
Search Tags:#optimization#llm#gpu#quantization#inference#nvidia#tensorrt-llm
One-click install
npx skills add https://github.com/heysuhas/hermes_cli --skill tensorrt-llm-heysuhas
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/heysuhas/hermes_cli/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/heysuhas/hermes_cli --skill tensorrt-llm-heysuhas

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This Skill addresses the high latency and low throughput challenges of deploying large language models in production, enabling 10-100x faster inference compared to standard PyTorch implementations.

Core Features & Use Cases

  • High-Performance Inference: Utilizes NVIDIA TensorRT to optimize models for maximum throughput and minimal latency.
  • Advanced Quantization: Supports FP8, INT4, and FP4 quantization to significantly reduce memory footprint and increase speed.
  • Production Scaling: Enables in-flight batching, multi-GPU tensor parallelism, and pipeline parallelism for large-scale model serving.
  • Use Case: Deploying a Llama 3-70B model on a multi-GPU cluster to serve real-time chat requests with sub-20ms token latency.

Quick Start

Use the tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with a batch size of 256 on four GPUs.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference latency for production environments on NVIDIA GPUs?▼

Optimizing LLM inference for NVIDIA GPUs requires compiling models with TensorRT to apply in-flight batching and tensor parallelism, achieving 10-100x faster speed compared to standard PyTorch implementations.

What's the best way to reduce the memory footprint of large language models during inference?▼

Reducing LLM memory footprint during inference is best achieved through quantization techniques like FP8, INT4, or FP4, which significantly decrease memory requirements while increasing processing speed.

How do I deploy a Llama 3 model on a multi-GPU cluster for real-time chat requests?▼

Deploying Llama 3 on a multi-GPU cluster involves configuring multi-node tensor parallelism and pipeline parallelism to serve real-time chat requests with sub-20ms token latency.

Does TensorRT-LLM in-flight batching work with standard PyTorch model implementations?▼

TensorRT-LLM in-flight batching is designed for NVIDIA GPU hardware optimization and does not work directly with standard PyTorch implementations, requiring conversion to the TensorRT engine format first.

What quantization formats does TensorRT-LLM support for high-throughput inference?▼

TensorRT-LLM supports FP8, INT4, and FP4 quantization formats for high-throughput inference, allowing flexible trade-offs between memory footprint reduction and model accuracy.

Related Skills

Explore More Software Engineering Skills→
arsityarsity

tensorrt-llm

Optimize LLM inference throughput and latency on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
hung-phanhung-phan
1

triton-inference-server

Serve ML models with dynamic batching and concurrent execution via Triton Inference Server.

Community
Advanced
superfhpsuperfhp

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints using vLLM.

Community
Advanced