skills.rest
twjarviszyra-web
twjarviszyra-webCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

3|Updated Apr 4, 2026
Search Tags:#quantization#autoscaling#nvidia#llm inference#tensorrt-llm#gpu serving#multi-gpu
One-click install
npx skills add https://github.com/twjarviszyra-web/hermes-unbound --skill tensorrt-llm-twjarviszyra-web
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/twjarviszyra-web/hermes-unbound/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/twjarviszyra-web/hermes-unbound --skill tensorrt-llm-twjarviszyra-web

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill helps you deploy and tune large language model inference on NVIDIA GPUs when you need much lower latency, higher throughput, and efficient GPU memory usage than a default PyTorch-based setup can provide.

Core Features & Use Cases

  • High-performance inference: Optimize model serving with TensorRT-LLM for fast generation on A100, H100, and similar NVIDIA hardware.
  • Production deployment: Configure OpenAI-compatible serving, health checks, metrics, batching, and autoscaling for real-world API workloads.
  • Multi-GPU scaling: Choose tensor parallelism, pipeline parallelism, or expert parallelism to fit large models across one or more nodes.
  • Quantization and memory tuning: Apply FP8 or INT4 strategies, paged KV cache, chunked context, and speculative decoding to improve speed and efficiency.
  • Use case: A team serving a 70B model in production can use this Skill to select the right parallelism, quantization, and batching settings for stable low-latency traffic.

Quick Start

Use the tensorrt-llm skill to recommend a production-ready NVIDIA GPU serving configuration for my model, including optimization, scaling, and monitoring settings.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for lower latency and higher throughput on NVIDIA GPUs?▼

Optimize LLM inference on NVIDIA GPUs by applying TensorRT-LLM to reduce latency and increase throughput. This involves configuring batching, paged KV cache management, and parallelism to accelerate text generation on A100 and H100 hardware.

What's the best way to deploy a 70B model across multiple GPUs in production?▼

Deploy a 70B model across multiple GPUs by selecting tensor parallelism, pipeline parallelism, or expert parallelism. This fits large models across one or more nodes, ensuring stable low-latency traffic for production serving workloads.

How does FP8 or INT4 quantization improve GPU memory usage during model serving?▼

FP8 or INT4 quantization improves GPU memory usage by reducing the precision of model weights during execution. Combined with chunked context and speculative decoding, these strategies significantly improve inference speed and memory efficiency.

Can I set up an OpenAI-compatible API for autoscaling LLM serving workloads?▼

Set up an OpenAI-compatible API for autoscaling LLM serving by configuring health checks, metrics, and batching. This provides a production-ready endpoint that handles real-world API workloads with automatic scaling adjustments.

Do I need CUDA-capable hardware to use TensorRT-LLM for inference optimization?▼

CUDA-capable NVIDIA hardware is required to use TensorRT-LLM for inference optimization. The framework specifically targets A100, H100, and similar NVIDIA GPUs to deliver fast generation and efficient memory usage for production deployments.

Related Skills

Explore More Software Engineering Skills→
ever-oliever-oli
3

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and CUDA graphs.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
cloudlinesscloudliness

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced