skills.rest
ovachiever
ovachieverCOMMUNITY
Explore all skills by this author →
Website𝕏@erikjfritschGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

52|6|Updated Nov 24, 2025
Search Tags:#LLM#GPU#inference#TensorRT#INT4#FP8#multi-GPU
One-click install
npx skills add https://github.com/ovachiever/droid-tings --skill tensorrt-llm
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/ovachiever/droid-tings/tree/main/skills/tensorrt-llm
Command: npx skills add https://github.com/ovachiever/droid-tings --skill tensorrt-llm

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt_llm, torch, and includes scripts (resource) and references (resource) and templates (resource) components.

What problem does it solve?

TensorRT-LLM optimizes inference for large language models on NVIDIA GPUs, delivering maximum throughput and minimal latency with quantization, in-flight batching, and multi-GPU scaling.

Core Features & Use Cases

  • FP8/INT4 quantization, in-flight batching, CUDA graphs
  • Tensor/pipeline/expert parallelism for scalable multi-GPU deployments
  • Production serving via trtllm-serve API

Quick Start

Install tensorrt_llm, instantiate an LLM with FP8 quantization, and run sample inferences; optionally deploy with trtllm-serve.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference speed on NVIDIA GPUs?▼

TensorRT-LLM optimizes LLM inference on NVIDIA GPUs by applying quantization (FP8/INT4), in-flight batching, and CUDA graphs to maximize throughput and minimize latency. It supports multi-GPU scaling and production serving via OpenAI-compatible APIs.

Can I use TensorRT-LLM for quantized model inference?▼

Yes. TensorRT-LLM supports FP8 and INT4 quantization for reduced memory footprint and faster inference. It handles automatic model loading, compilation, and quantization configuration for both single and multi-GPU deployments.

What's the best way to deploy LLM inference at scale across multiple GPUs?▼

TensorRT-LLM enables multi-GPU and multi-node scaling through tensor, pipeline, and expert parallelism. Combined with in-flight batching and KV cache optimization, it delivers production-grade throughput and latency for high-demand deployments.

Does TensorRT-LLM work with existing PyTorch models?▼

TensorRT-LLM integrates with PyTorch workflows through its dependencies on torch and tensorrt_llm. It compiles PyTorch models to optimized TensorRT engines, applying quantization and parallelism strategies without requiring model rewrites.

What advanced features does TensorRT-LLM provide for production inference?▼

TensorRT-LLM includes speculative decoding, CUDA graph optimization, KV cache tuning, and OpenAI-compatible API serving with built-in monitoring. These features reduce latency, improve resource utilization, and enable drop-in deployment for existing LLM applications.

Related Skills

Explore More Software Engineering Skills→
informatico-madridinformatico-madrid
1

tensorrt-llm

Optimizes LLM serving on NVIDIA hardware with batched GPU inference.

Community
Advanced
Chris-Chai-MinjaeChris-Chai-Minjae

tensorrt-llm

Accelerate large language model inference on NVIDIA GPUs with FP8/INT4 quantization.

Community
Advanced
tadod12tadod12

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for production GPU serving.

Community
Advanced
zhouboyu-xrealzhouboyu-xreal

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

Community
Advanced