skills.rest
Rawgrowth-Consulting
Rawgrowth-ConsultingOFFICIAL
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

Updated Apr 23, 2026
Search Tags:#optimization#llm#gpu#inference#serving#tensorrt#multigpu
One-click install
npx skills add https://github.com/Rawgrowth-Consulting/rawclaw-agent --skill tensorrt-llm-rawgrowth-consulting
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Rawgrowth-Consulting/rawclaw-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/Rawgrowth-Consulting/rawclaw-agent --skill tensorrt-llm-rawgrowth-consulting

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100/H100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8/INT4), in-flight batching, and multi-GPU scaling.

Core Features & Use Cases

  • Performance optimizations: In-flight batching, quantization (FP8/INT4), CUDA graphs, and tensor/pipeline/expert parallelism for multi-GPU deployment.
  • Production-grade serving: OpenAI-compatible API with built-in metrics, health checks, and multi-node deployment guidance.
  • Use Case: Deploy a large LLM on a cluster of GPUs to serve low-latency queries with quantized models and batched requests.

Quick Start

Install the tensorrt_llm package and start the server with your target model to begin serving.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I speed up LLM inference for production deployment on NVIDIA GPUs?▼

Speed up LLM inference by leveraging NVIDIA TensorRT to achieve high throughput and low latency. It targets production deployments on A100/H100 GPUs, supporting FP8/INT4 quantization, in-flight batching, and multi-GPU scaling across single or multiple nodes.

What is the best way to serve large language models with low latency and batched requests?▼

Serving large language models with low latency is best handled via in-flight batching and CUDA graphs. This approach provides an OpenAI-compatible API with built-in metrics and health checks for production-grade multi-node deployment.

Can I use FP8 or INT4 quantization for multi-GPU LLM inference?▼

Yes, you can use FP8 or INT4 quantization for multi-GPU LLM inference. This supports tensor, pipeline, and expert parallelism to scale large model deployment efficiently across single or multiple nodes.

Does TensorRT LLM inference provide an OpenAI-compatible API with monitoring?▼

TensorRT LLM inference provides an OpenAI-compatible API with built-in metrics and health checks. It satisfies production-grade serving requirements including automatic model compilation and multi-GPU orchestration.

Why use TensorRT for LLM inference over standard PyTorch?▼

Use TensorRT for LLM inference to achieve 10-100x faster inference than standard PyTorch. It leverages NVIDIA GPU hardware optimizations like in-flight batching, CUDA graphs, and FP8 quantization for maximum throughput.

Do I need A100 or H100 GPUs for TensorRT LLM inference?▼

You need A100 or H100 NVIDIA GPUs for optimal TensorRT LLM inference. The system targets production deployments on these specific hardware architectures to fully leverage FP8 quantization and multi-GPU scaling.

Related Skills

Explore More Software Engineering Skills→
JuanMS20JuanMS20

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
TehminanazTehminanaz
1

openai-agents-sdk-groq-integration

Integrate the OpenAI Agents SDK with Groq's endpoint in FastAPI applications.

Community
Advanced
OpenLAIROpenLAIR
1.0k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and multi-GPU scaling.

Official
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced