skills.rest
thisismynewfmail-ui
thisismynewfmail-uiCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference with NVIDIA TensorRT-LLM on A100/H100 GPUs.

Updated Apr 12, 2026
Search Tags:#batching#quantization#inference#fp8#nvidia#tensorrt-llm#multi-gpu
One-click install
npx skills add https://github.com/thisismynewfmail-ui/Monika-agent --skill tensorrt-llm-thisismynewfmail-ui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/thisismynewfmail-ui/Monika-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/thisismynewfmail-ui/Monika-agent --skill tensorrt-llm-thisismynewfmail-ui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the friction of deploying and running large language models at production scale on NVIDIA GPUs by optimizing inference for maximum throughput and minimal latency, reducing cost and memory footprint through quantization and advanced scheduling.

Core Features & Use Cases

  • High-performance inference: Compile and serve LLMs with TensorRT-LLM for 10-100× speedups versus standard PyTorch inference.
  • Quantized and scalable serving: Support FP8, INT4, tensor/pipeline parallelism, in-flight batching, and multi-GPU or multi-node deployments for large models.
  • Production tooling: OpenAI-compatible server (trtllm-serve), Prometheus metrics, Kubernetes manifests, and autoscaling guidance for reliable production operation.
  • Use Case: Serve a Llama 3-70B chat model across 4 A100 GPUs with FP8 quantization to achieve low-latency interactive chat while maximizing tokens/sec and minimizing cloud cost.

Quick Start

Start the trtllm-serve server for meta-llama/Meta-Llama-3-8B with --tp_size 4 and --dtype fp8 to enable compiled, high-throughput inference.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for high throughput on NVIDIA GPUs?▼

Optimize LLM inference by compiling models with TensorRT-LLM to achieve 10-100× speedups over standard PyTorch, specifically targeting NVIDIA A100 and H100 GPUs to maximize tokens per second. This reduces latency and cloud costs for production deployments.

How do I deploy a Llama 3 70B model across multiple GPUs with FP8 quantization?▼

Deploy Llama 3-70B across multiple GPUs using TensorRT-LLM with tensor parallelism and FP8 quantization. This configuration enables low-latency interactive chat while maximizing throughput and minimizing the memory footprint on A100 GPUs.

Does TensorRT-LLM support in-flight batching and OpenAI-compatible serving?▼

TensorRT-LLM supports in-flight batching for dynamic request scheduling and provides an OpenAI-compatible server via trtllm-serve. This allows drop-in replacement for standard OpenAI API endpoints while delivering optimized inference performance.

What is the best way to scale LLM inference and monitor performance in production?▼

Scale LLM inference using multi-GPU and multi-node deployments with tensor and pipeline parallelism. Monitor production performance using integrated Prometheus metrics, and manage reliability with provided Kubernetes manifests and autoscaling guidance.

Can I use INT4 quantization to reduce memory footprint for large language models?▼

INT4 quantization is supported alongside FP8 to significantly reduce the memory footprint of large language models. Applying these quantization techniques allows serving larger models within constrained GPU memory limits while maintaining high throughput.

Why choose TensorRT-LLM over standard PyTorch for LLM serving?▼

TensorRT-LLM compiles LLMs into highly optimized engines, delivering 10-100× speedups compared to standard PyTorch inference. It is purpose-built for NVIDIA GPUs, offering advanced scheduling, in-flight batching, and quantization not natively available in PyTorch.

Related Skills

Explore More Software Engineering Skills→
richardnguyen0715richardnguyen0715

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
frankxaifrankxai
2

Kubernetes AI Expert

Deploys AI workloads on Kubernetes with GPU scheduling and scalable model serving via manifests.yaml.

Community
Advanced
JamesFincherJamesFincher

tensorrt-llm

Optimizes LLM inference serving on NVIDIA GPUs using TensorRT-LLM with quantization and parallelism options.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced