skills.rest
jason660519
jason660519COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Deploy LLM inference on NVIDIA GPUs with quantization and parallelism.

Updated May 11, 2026
Search Tags:#deployment#optimization#llm#gpu#inference#tensorrt#multi-gpu
One-click install
npx skills add https://github.com/jason660519/Project-Manager --skill tensorrt-llm-jason660519
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/jason660519/Project-Manager/tree/main/hermes-agent/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/jason660519/Project-Manager --skill tensorrt-llm-jason660519

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference deployment on NVIDIA GPUs to maximize throughput and minimize latency.

Core Features & Use Cases

  • In-flight batching: dynamically batches requests for higher throughput.
  • Quantization: FP8/INT4 to reduce memory and improve speed.
  • Parallelism: Tensor and pipeline parallelism for multi-GPU and multi-node setups.
  • Production Serving: OpenAI-compatible API, Prometheus metrics, and scalable deployment.

Quick Start

Install the tensorrt_llm package and launch the trtllm-serve daemon to deploy a model on NVIDIA GPUs.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference on NVIDIA GPUs for production workloads?▼

Accelerate LLM inference on NVIDIA GPUs using TensorRT-LLM to apply in-flight batching, FP8/INT4 quantization, and CUDA-graph acceleration, maximizing throughput and minimizing latency for production AI workloads.

What is the best way to configure multi-GPU parallelism for LLM deployment?▼

Configure multi-GPU LLM deployment by setting up tensor parallelism, pipeline parallelism, and expert parallelism strategies to scale inference across single-node and multi-node clusters efficiently.

How does in-flight batching improve LLM inference throughput?▼

In-flight batching improves LLM inference throughput by dynamically batching incoming requests during execution, allowing the engine to process concurrent requests efficiently without waiting for static batch boundaries.

Can I deploy an OpenAI-compatible API for LLM serving with Prometheus metrics?▼

Yes, you can deploy an OpenAI-compatible API for LLM serving with Prometheus metrics by launching the trtllm-serve daemon, which enables scalable production serving and performance monitoring.

Do I need FP8 or INT4 quantization to reduce LLM memory usage?▼

You need FP8 or INT4 quantization to reduce LLM memory usage and improve inference speed, configuring these quantization settings through the cohesive API to optimize model loading and deployment.

When should I use TensorRT-LLM over other LLM inference optimization tools?▼

Use TensorRT-LLM over other LLM inference optimization tools when you require scalable multi-GPU setups, CUDA-graph acceleration, and production-grade serving features for high-throughput NVIDIA GPU deployments.

Related Skills

Explore More Software Engineering Skills→
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
hungthinh04hungthinh04

serving-llms-vllm

Serve LLMs with OpenAI-compatible endpoints and optimized inference.

Community
Advanced
masermediagroup-stackmasermediagroup-stack
1

model-serving

Deploy and manage LLM and ML model-serving stacks with optimized backends.

Community
Advanced
Dq666dingerDq666dinger
1

serving-llms-vllm

Serve LLMs via an OpenAI-compatible API using vLLM with quantization and tensor parallelism.

Community
Advanced