tensorrt-llm

Name: tensorrt-llm
Availability: InStock
Author: daddyelonmusk69

Accelerate LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated Apr 12, 2026

Search Tags:#deployment #llm #gpu #quantization #inference #tensorrt

Ready to run

npx skills add https://github.com/DaddyElonMusk69/motis-agent --skill tensorrt-llm-daddyelonmusk69

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/DaddyElonMusk69/motis-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/DaddyElonMusk69/motis-agent --skill tensorrt-llm-daddyelonmusk69

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill addresses the difficulty of achieving production‑grade LLM inference speed on NVIDIA GPUs, where standard PyTorch pipelines are too slow and costly for real‑time or high‑throughput applications.

Core Features & Use Cases

GPU‑accelerated serving: Leverages TensorRT‑LLM to deliver up to 24,000 tokens‑per‑second on an H100, dramatically reducing latency.
Quantization and batching: Supports FP8/INT4 quantization and in‑flight batching for further speed and memory gains.
Multi‑GPU scaling: Provides guidance for tensor, pipeline, and expert parallelism to run models from 8B to 400B parameters across single or multiple nodes.

Use case example: A trading firm deploys Meta‑Llama‑3‑8B with TensorRT‑LLM on a four‑GPU server to generate market summaries within milliseconds, enabling real‑time decision making.

Quick Start

Ask the skill to configure and launch TensorRT‑LLM serving for a chosen model on your GPU cluster.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I accelerate LLM inference on NVIDIA GPUs for production serving?▼

Use TensorRT to accelerate LLM inference on NVIDIA GPUs, replacing slow PyTorch pipelines to achieve up to 24,000 tokens-per-second on an H100 for high-throughput serving.

What is the best way to run large language models like Meta-Llama-3 across multiple GPUs?▼

Multi-GPU scaling for large language models like Meta-Llama-3 uses tensor, pipeline, and expert parallelism to distribute 8B to 400B parameter models across single or multiple nodes.

How does quantization improve LLM inference speed and memory usage?▼

Quantization improves LLM inference speed and memory usage by applying FP8 or INT4 formats, reducing model footprint and enabling higher throughput on NVIDIA GPUs.

Can I use in-flight batching to increase throughput for real-time LLM applications?▼

In-flight batching increases throughput for real-time LLM applications by dynamically scheduling incoming requests, enabling higher token generation rates and reducing latency for production deployments.

What CUDA version and GPU hardware do I need for TensorRT-LLM deployment?▼

Deployment requires NVIDIA TensorRT, a compatible CUDA version, and appropriate GPU hardware to meet performance and memory constraints for large language model serving.

Why is standard PyTorch too slow for real-time LLM serving?▼

Standard PyTorch pipelines are too slow for real-time LLM serving because unoptimized execution incurs high latency and computational costs, making them unsuitable for high-throughput, millisecond-level response applications.