skills.rest
kaminocorp
kaminocorpOFFICIAL
Explore all skills by this author →
Website𝕏@kaminocorpoGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

228|35|Updated Mar 9, 2026
Search Tags:#latency#gpu#throughput#nvidia#inference-optimization#tensorrt-llm
One-click install
npx skills add https://github.com/kaminocorp/hermes-alpha --skill tensorrt-llm-kaminocorp
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/kaminocorp/hermes-alpha/tree/main/hermes-agent/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/kaminocorp/hermes-alpha --skill tensorrt-llm-kaminocorp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimizes LLM inference with NVIDIA TensorRT to deliver maximum throughput and minimum latency in production environments on CUDA-enabled GPUs.

Core Features & Use Cases

  • Performance optimizations such as in-flight batching, paged KV cache, CUDA graphs to reduce kernel launch overhead.
  • Parallelism and scaling across GPUs and nodes via tensor, pipeline, and expert parallelism.
  • Quantization support (FP8, INT4, FP4) to boost throughput with minimal accuracy loss.
  • Production serving through trtllm-serve with an OpenAI-compatible API, metrics, and multi-node deployment.

Quick Start

Install the tensorrt-llm package and start the server with a target model to bootstrap high-throughput inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize large language model inference on NVIDIA GPUs for production?▼

To optimize large language model inference on NVIDIA GPUs, apply in-flight batching, paged KV cache, and CUDA graphs to reduce kernel launch overhead and maximize throughput. TensorRT-LLM provides these performance optimizations specifically for CUDA-enabled production environments.

What is the best way to scale LLM serving across multiple GPUs and nodes?▼

The best way to scale LLM serving across multiple GPUs and nodes is using tensor, pipeline, and expert parallelism. TensorRT-LLM supports these parallelism strategies to distribute inference workloads efficiently across multi-node setups.

Can I use FP8 or INT4 quantization to boost LLM inference throughput?▼

Yes, you can use FP8, INT4, or FP4 quantization to boost LLM inference throughput with minimal accuracy loss. TensorRT-LLM provides quantization support to accelerate model serving while maintaining output quality.

How do I start serving an LLM with an OpenAI-compatible API?▼

To start serving an LLM with an OpenAI-compatible API, install the tensorrt-llm package and use the trtllm-serve server tool. This provides high-throughput inference serving with metrics and multi-node deployment support.

Do I need CUDA and specific libraries to run TensorRT-LLM optimizations?▼

Yes, you need CUDA and the core TensorRT-LLM library to run inference optimizations on NVIDIA GPUs. The setup requires compatible models and CUDA-enabled GPU hardware to maximize throughput and minimize latency.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
dawsonblockdawsonblock

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM on multi-GPU servers.

Community
Advanced
OpenLAIROpenLAIR
1.0k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and multi-GPU scaling.

Official
Advanced
BermudaLocalsBermudaLocals
1

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs using quantization and multi-GPU scaling.

Community
Advanced