skills.rest
1thirteeng3
1thirteeng3COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Updated Mar 24, 2026
Search Tags:#llm#gpu#quantization#inference#tensorrt#multigpu
One-click install
npx skills add https://github.com/1thirteeng3/greenmoire --skill tensorrt-llm-1thirteeng3
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/1thirteeng3/greenmoire/tree/main/integrations/hermes-agent/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/1thirteeng3/greenmoire --skill tensorrt-llm-1thirteeng3

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TensorRT-LLM accelerates large language model inference on NVIDIA GPUs, delivering higher throughput and lower latency for production deployments.

Core Features & Use Cases

  • In-flight batching, FP8/INT4 quantization, and multi-GPU scaling for high-throughput serving.
  • Production-grade deployment on A100/H100 hardware with TensorRT-LLM to achieve maximum efficiency and scalability.
  • Typical use cases include real-time chat completions, offline batch inference, and large-scale inference pipelines in ML ops workflows.

Quick Start

Deploy a TensorRT-LLM model on an NVIDIA GPU cluster and start a fast, scalable inference server.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I speed up LLM inference on NVIDIA GPUs for production deployments?▼

To speed up LLM inference on NVIDIA GPUs, leverage TensorRT-LLM to maximize throughput and minimize latency. It enables in-flight batching and multi-GPU scaling specifically for A100 and H100 hardware.

Does TensorRT-LLM support FP8 and INT4 quantization for large language models?▼

Yes, TensorRT-LLM supports FP8 and INT4 quantization. These techniques reduce memory footprint and accelerate computation during LLM inference on NVIDIA GPUs without significantly degrading model accuracy.

What is in-flight batching and how does it optimize real-time chat completions?▼

In-flight batching optimizes real-time chat completions by dynamically scheduling incoming requests. This maximizes GPU utilization and increases throughput for LLM inference servers handling concurrent user interactions.

Can I use the tensorrt_llm package with Torch for scalable multi-node deployment?▼

Yes, you can use the tensorrt_llm package with Torch to achieve scalable multi-node deployment. This integration allows you to distribute LLM inference workloads efficiently across multiple NVIDIA GPUs.

When do I need multi-GPU scaling for offline batch inference?▼

You need multi-GPU scaling for offline batch inference when processing massive datasets requires more memory or compute than a single A100 or H100 GPU can provide, ensuring high-throughput pipeline execution.

Related Skills

Explore More Software Engineering Skills→
brittaniebuffiecsubrittaniebuffiecsu
1

tensorrt-llm

Optimizes LLM inference on NVIDIA GPUs using TensorRT with FP8/INT4 quantization and multi-GPU scaling.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
matthew-johnsonmatthew-johnson

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

Community
Advanced
zeronx798zeronx798

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM on A100 and H100 GPUs.

Community
Advanced