skills.rest
sheawinkler
sheawinklerCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Accelerate LLM inference on NVIDIA GPUs with FP8/INT4 quantization and multi-GPU scaling.

78|16|Updated Apr 23, 2026
Search Tags:#production#llm#gpu#quantization#inference#tensorrt#multigpu
One-click install
npx skills add https://github.com/sheawinkler/hermes-agent-ultra --skill tensorrt-llm-sheawinkler
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/sheawinkler/hermes-agent-ultra/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/sheawinkler/hermes-agent-ultra --skill tensorrt-llm-sheawinkler

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates and optimizes LLM inference on NVIDIA GPUs, enabling production-grade, low-latency deployments with FP8/INT4 quantization and multi-GPU scalability.

Core Features & Use Cases

  • FP8/INT4 quantization for faster inference and reduced memory footprint.
  • Multi-GPU parallelism (TP/PP/EP) for large models and high throughput.
  • Production-ready serving with an OpenAI-compatible API, metrics, health checks, and monitoring.
  • Use cases include deploying enterprise-scale LLMs requiring predictable latency and reliability.

Quick Start

Start the TensorRT-LLM server with a model and basic configuration to begin serving requests.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production?▼

To accelerate LLM inference on NVIDIA GPUs for production, you can apply GPU-accelerated optimization techniques like FP8/INT4 quantization and in-flight batching to reduce latency and memory footprint. This enables enterprise-scale deployments with predictable throughput.

What is the best way to scale large language models across multiple GPUs?▼

Scaling large language models across multiple GPUs requires multi-GPU parallelism, utilizing Tensor Parallelism (TP), Pipeline Parallelism (PP), and Expert Parallelism (EP). These strategies distribute model execution to achieve high throughput for enterprise-scale models.

Does TensorRT-LLM support serving OpenAI-compatible APIs?▼

Yes, TensorRT-LLM supports production-ready serving with an OpenAI-compatible API. It includes built-in metrics, health checks, and monitoring to ensure reliable deployment for enterprise-scale LLMs requiring predictable latency.

When do I need INT4 or FP8 quantization for LLM inference?▼

You need INT4 or FP8 quantization for LLM inference when aiming for faster processing speeds and a reduced memory footprint. It is essential for deploying enterprise-scale models that require predictable latency and high throughput on NVIDIA GPUs.

Can I use in-flight batching to improve LLM throughput?▼

Yes, you can use in-flight batching to improve LLM throughput. It dynamically processes incoming requests alongside currently generating ones, maximizing GPU utilization and ensuring low-latency production deployment for high-volume inference workloads.

Why does deploying large LLMs require multi-GPU parallelism?▼

Deploying large LLMs requires multi-GPU parallelism because single-device memory limits model size and throughput. Using TP, PP, and EP distributes execution across NVIDIA GPUs, enabling production-grade serving of enterprise-scale models.

Related Skills

Explore More Software Engineering Skills→
richardnguyen0715richardnguyen0715

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
TehminanazTehminanaz
1

openai-agents-sdk-groq-integration

Integrate the OpenAI Agents SDK with Groq's endpoint in FastAPI applications.

Community
Advanced
mahamaneharouna9-cpumahamaneharouna9-cpu

monitoring-expert

Configure Pino logging, Sentry error tracking, health checks, and Prometheus metrics for Node.js applications.

Community
Advanced
davpatel605-beepdavpatel605-beep

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced