skills.rest
davpatel605-beep
davpatel605-beepCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Updated Jun 25, 2026
Search Tags:#performance optimization#quantization#model serving#llm inference#tensorrt-llm#nvidia gpu#multi-gpu
One-click install
npx skills add https://github.com/davpatel605-beep/hermusagent --skill tensorrt-llm-davpatel605-beep
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/davpatel605-beep/hermusagent/tree/main/backend/vendor/hermes/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/davpatel605-beep/hermusagent --skill tensorrt-llm-davpatel605-beep

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This Skill helps production teams accelerate large language model inference by optimizing deployments for NVIDIA GPUs with TensorRT-LLM, reducing latency and increasing throughput.

Core Features & Use Cases

  • Inference Optimization: Applies TensorRT-LLM techniques such as quantization, in-flight batching, paged KV cache, and CUDA graph optimization for faster generation.
  • Production Serving: Supports OpenAI-compatible serving, monitoring, scaling, and deployment patterns for enterprise LLM applications.
  • Multi-GPU Scaling: Guides tensor parallelism, pipeline parallelism, and distributed deployment for large models on A100, H100, and similar NVIDIA hardware.
  • Use Case: Deploy a high-traffic LLM API on NVIDIA GPUs with FP8 optimization, batching, and multi-GPU parallelism to maximize tokens per second.

Quick Start

Use the tensorrt-llm skill to optimize and deploy my LLM inference service on NVIDIA GPUs with low latency and high throughput.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference latency and throughput on NVIDIA GPUs?▼

Optimize LLM inference latency and throughput on NVIDIA GPUs by applying TensorRT-LLM techniques like FP8 quantization, in-flight batching, paged KV cache, and CUDA graph optimization to maximize tokens per second.

What is the best way to scale large language model serving across multiple NVIDIA GPUs?▼

Scale large language model serving across multiple NVIDIA GPUs by configuring tensor parallelism and pipeline parallelism within TensorRT-LLM, enabling distributed deployment for large models on A100 and H100 hardware.

Does TensorRT-LLM support OpenAI-compatible API serving for production deployments?▼

TensorRT-LLM supports OpenAI-compatible API serving for production deployments, providing enterprise patterns for monitoring, scaling, and high-traffic LLM application integration.

How do I apply FP8 quantization to reduce GPU memory usage during LLM inference?▼

Apply FP8 quantization to reduce GPU memory usage during LLM inference by utilizing TensorRT-LLM optimization features, which lower memory bandwidth requirements and accelerate generation throughput.

When do I need in-flight batching and paged KV cache for LLM inference optimization?▼

You need in-flight batching and paged KV cache for LLM inference optimization when handling high-traffic API workloads, as these TensorRT-LLM features dynamically manage requests to minimize generation latency.

Can I use torch models with TensorRT-LLM for production inference deployment?▼

You can use torch models with TensorRT-LLM for production inference deployment by converting them into optimized engines that leverage NVIDIA GPU hardware capabilities for low-latency serving.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
LancerLabLancerLab

perf-nsight-compute-analysis

Analyze NVIDIA Nsight Compute reports to classify GPU kernel bottlenecks.

Official
Intermediate
vasilyu1983vasilyu1983
73

ai-llm-inference

Optimize LLM inference performance and cost with vLLM and quantization.

Community
Advanced