skills.rest
matthew-johnson
matthew-johnsonCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

Updated Apr 20, 2026
Search Tags:#production#llm#gpu#quantization#inference#tensorRT
One-click install
npx skills add https://github.com/matthew-johnson/hermes-agent --skill tensorrt-llm-matthew-johnson
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/matthew-johnson/hermes-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/matthew-johnson/hermes-agent --skill tensorrt-llm-matthew-johnson

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates production-grade LLM inference by leveraging NVIDIA TensorRT to deliver high throughput and low latency across single and multi-GPU deployments.

Core Features & Use Cases

  • Inference optimizations: FP8/INT4 quantization, in-flight batching, CUDA graphs, and KV caching for throughput at scale
  • Multi-GPU and multi-node deployment: tensor, pipeline, and expert parallelism for large models
  • Production-ready serving: OpenAI-compatible API with monitoring, metrics, and autoscaling support

Quick Start

Start a production deployment by running trtllm-serve with your chosen model and TP/quantization settings.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs in production?▼

To maximize LLM inference throughput on NVIDIA GPUs, apply TensorRT optimizations like FP8 or INT4 quantization, in-flight batching, and KV caching. These techniques significantly reduce latency and increase token generation rates for production deployments on A100 and H100 clusters.

What is the best way to serve large language models with an OpenAI-compatible API?▼

Serving large language models with an OpenAI-compatible API is best achieved using optimized inference engines like TensorRT-LLM. It provides a production-ready serving layer with built-in monitoring, metrics, and autoscaling support, ensuring high throughput and low latency for API requests.

Can I scale LLM inference across multiple GPUs and nodes using tensor parallelism?▼

Yes, you can scale LLM inference across multiple GPUs and nodes using tensor, pipeline, and expert parallelism. This multi-GPU and multi-node deployment capability allows you to run large models that exceed single-GPU memory limits while maintaining high inference throughput.

Does FP8 and INT4 quantization work for deploying LLMs on A100 and H100 GPUs?▼

FP8 and INT4 quantization work effectively for deploying LLMs on A100 and H100 GPUs. These quantization methods reduce memory footprint and computational overhead, enabling faster inference and higher throughput without significant loss in model accuracy.

How do I start a production deployment with TensorRT-LLM?▼

To start a production deployment with TensorRT-LLM, run the trtllm-serve command with your chosen model, tensor parallelism settings, and quantization configurations. This enables immediate serving with optimized CUDA graphs and in-flight batching.

Why should I use in-flight batching and KV caching for LLM serving?▼

You should use in-flight batching and KV caching for LLM serving because they dynamically manage continuous execution streams and store previous token states. This minimizes idle GPU time and drastically improves overall inference throughput under heavy concurrent request loads.

Related Skills

Explore More Software Engineering Skills→
richardnguyen0715richardnguyen0715

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
hung-phanhung-phan
1

ray-serve

Deploy ML models to Ray Serve with autoscaling and request batching.

Community
Advanced
cryptopaficryptopafi

serving-llms-vllm

Serve large language models with high-throughput inference for OpenAI-compatible endpoints.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced