skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs using TensorRT-LLM with quantization and batching.

11.5k|842|Updated Nov 3, 2025
Search Tags:#quantization#production serving#inference optimization#llm inference#tensorrt-llm#nvidia gpu
One-click install
npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill tensorrt-llm-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/Orchestra-Research/AI-Research-SKILLs --skill tensorrt-llm-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill optimizes Large Language Model (LLM) inference for maximum throughput and minimal latency on NVIDIA GPUs, making production deployment of LLMs significantly faster and more efficient.

Core Features & Use Cases

  • High-Performance Inference: Achieves 10-100x faster inference speeds compared to standard frameworks like PyTorch.
  • Production Deployment: Ideal for serving LLMs in production environments on NVIDIA hardware (A100/H100/GB200).
  • Advanced Optimizations: Supports quantization (FP8, INT4), in-flight batching, multi-GPU scaling, and more.
  • Use Case: Deploying a Llama 3-70B model for a real-time chatbot application where low latency and high concurrent user handling are critical.

Quick Start

Use the tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with tensor parallelism across 4 GPUs.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for production serving on NVIDIA GPUs?▼

TensorRT-LLM accelerates LLM inference on NVIDIA GPUs, achieving 10-100x faster speeds compared to standard PyTorch. It is designed for high throughput and low latency in production serving scenarios.

What is the best way to deploy a Llama 3 model with low latency for real-time chatbots?▼

Deploying Llama 3 models with TensorRT-LLM enables real-time chatbot performance by utilizing tensor parallelism across multiple GPUs. This approach handles high concurrent users while maintaining critical low latency.

Can I use FP8 or INT4 quantization to reduce LLM inference memory usage?▼

Yes, TensorRT-LLM supports FP8 and INT4 quantization to reduce memory usage and accelerate inference. These advanced optimizations maximize efficiency when deploying large language models on NVIDIA hardware.

Does TensorRT-LLM support multi-GPU parallelism for large language models?▼

Yes, TensorRT-LLM supports multi-GPU parallelism, including tensor parallelism across multiple GPUs. This allows you to scale large language model inference workloads efficiently across NVIDIA hardware.

How much faster is TensorRT-LLM compared to standard PyTorch for LLM inference?▼

TensorRT-LLM achieves 10-100x faster inference speeds compared to standard frameworks like PyTorch. This substantial performance gain is crucial for production serving scenarios requiring high throughput and low latency.

Related Skills

Explore More Software Engineering Skills→
JuanMS20JuanMS20

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
automatedigitalautomatedigital
1

serving-llms-vllm

Serve large language models with vLLM's optimized inference engine.

Official
Advanced
cloudlinesscloudliness

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
peteromalletpeteromallet
97

serving-llms-vllm

Deploy LLM inference with vLLM using OpenAI-compatible endpoints, quantization, and tensor parallelism.

Community
Intermediate