skills.rest
Chris-Chai-Minjae
Chris-Chai-MinjaeCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Accelerate large language model inference on NVIDIA GPUs with FP8/INT4 quantization.

Updated Apr 23, 2026
Search Tags:#optimization#llm#gpu#inference#serving#tensorrt#multigpu
One-click install
npx skills add https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge --skill tensorrt-llm-chris-chai-minjae
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/Chris-Chai-Minjae/hermes-agent-r1-bridge --skill tensorrt-llm-chris-chai-minjae

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates large language model inference on NVIDIA GPUs by delivering higher throughput and lower latency in production deployments.

Core Features & Use Cases

  • High-throughput inference on NVIDIA GPUs (A100/H100) with FP8/INT4 quantization.
  • Multi-GPU and multi-node deployment with tensor/pipeline/expert parallelism.
  • Production-grade serving with automatic model compilation, OpenAI-compatible API, and metrics support.

Quick Start

Install the tensorrt-llm package and run trtllm-serve with your model and parallelism settings.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate large language model inference on NVIDIA GPUs?▼

TensorRT-LLM accelerates LLM inference on NVIDIA GPUs by delivering higher throughput and lower latency in production deployments through automatic model compilation and quantization.

Can I deploy multi-GPU and multi-node LLM serving with tensor parallelism?▼

Yes, multi-GPU and multi-node LLM serving is supported using tensor, pipeline, and expert parallelism strategies to scale inference across multiple NVIDIA GPUs efficiently.

Does TensorRT-LLM support FP8 and INT4 quantization for inference?▼

FP8 and INT4 quantization are supported for high-throughput inference on NVIDIA A100 and H100 GPUs to reduce memory footprint and improve processing efficiency.

How do I start serving models with an OpenAI-compatible API?▼

Install the tensorrt-llm package and run trtllm-serve with your model and parallelism settings to deploy production-grade serving with an OpenAI-compatible API.

Does LLM serving with TensorRT-LLM provide Prometheus-compatible metrics?▼

Yes, production-grade LLM serving with TensorRT-LLM provides Prometheus-compatible metrics for monitoring inference workloads across multi-GPU and multi-node deployments.

Related Skills

Explore More Software Engineering Skills→
BermudaLocalsBermudaLocals
1

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs using quantization and multi-GPU scaling.

Community
Advanced
qcmuuqcmuu

serving-llms-vllm

Deploy vLLM inference endpoints with quantization and tensor parallelism.

Community
Advanced
moltis-orgmoltis-org
2.8k

serving-llms-vllm

Deploy a vLLM-based LLM serving stack with OpenAI-compatible endpoints.

Official
Advanced
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced