skills.rest
tadod12
tadod12COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for production GPU serving.

Updated Mar 18, 2026
Search Tags:#quantization#production-deployment#gpu-acceleration#inference-serving#fp8#tensorrt-llm#multi-gpu
One-click install
npx skills add https://github.com/tadod12/fraud-detection-research --skill tensorrt-llm-tadod12
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/tadod12/fraud-detection-research/tree/main/.agent/skills/12-inference-serving/tensorrt-llm
Command: npx skills add https://github.com/tadod12/fraud-detection-research --skill tensorrt-llm-tadod12

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates large language model inference by bridging the model runtime with NVIDIA TensorRT to deliver higher throughput and lower latency in production environments.

Core Features & Use Cases

  • Performance optimizations: In-flight batching, Flash Attention, FP8/INT4 quantization, CUDA graphs
  • Parallelism options: Tensor parallelism, Pipeline parallelism, Expert parallelism, multi-node deployments
  • Production serving: OpenAI-compatible API via trtllm-serve, Python API, metrics, health checks

Quick Start

Install the required dependencies and start serving with trtllm-serve using a supported model.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs?▼

Maximize LLM inference throughput on NVIDIA GPUs by leveraging TensorRT to apply in-flight batching, Flash Attention, and CUDA graphs for lower latency in production environments.

Does TensorRT-LLM support FP8 and INT4 quantization for production deployment?▼

Yes, production deployment supports FP8 and INT4 quantization to accelerate large language model inference, reducing memory footprint while maintaining high throughput on supported NVIDIA GPUs.

Can I scale LLM serving across multiple GPUs and nodes?▼

Scale LLM serving across multiple GPUs and nodes by configuring tensor parallelism, pipeline parallelism, and expert parallelism to distribute workloads efficiently over single or multi-node clusters.

How do I start serving a model using the OpenAI-compatible API?▼

Start serving a model using the OpenAI-compatible API by installing required dependencies and launching the trtllm-serve command with a supported model, utilizing built-in metrics and health checks.

Do I need CUDA-enabled GPUs to use TensorRT-LLM?▼

Yes, CUDA-enabled GPUs are strictly required, specifically optimized for NVIDIA A100 and H100 architectures to execute the TensorRT runtime and achieve maximum inference acceleration.

Related Skills

Explore More Software Engineering Skills→
Monjyu1101Monjyu1101
1

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
Clay-HHKClay-HHK
2

training-llms-megatron

Train multi-billion parameter language models with distributed parallelism on GPU clusters.

Community
Advanced
sheawinklersheawinkler
78

tensorrt-llm

Accelerate LLM inference on NVIDIA GPUs with FP8/INT4 quantization and multi-GPU scaling.

Community
Advanced