skills.rest
ragnarokhaa
ragnarokhaaCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT and quantization.

4|Updated Apr 19, 2026
Search Tags:#low latency#nvidia#inference optimization#high throughput#tensorrt#llm inference
One-click install
npx skills add https://github.com/ragnarokhaa/hermes --skill tensorrt-llm-ragnarokhaa
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/ragnarokhaa/hermes/tree/main/hermes-cerul-tech-news-package/hermes-cerul-tech-news-package/hermes-agent/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/ragnarokhaa/hermes --skill tensorrt-llm-ragnarokhaa

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill provides optimized inference for LLMs using NVIDIA TensorRT, offering significant speedups and reduced latency compared to standard PyTorch implementations.

Core Features & Use Cases

  • Inference Serving: Delivers high-throughput, low-latency LLM inference suitable for production deployment.
  • Quantization: Supports FP8 and INT4 quantization for faster inference and reduced memory usage.
  • Batch Inference: Enables efficient processing of multiple queries simultaneously, enhancing throughput.
  • Multi-GPU Support: Optimizes LLM inference across multiple GPUs, scaling to meet high demand.
  • Use Case: Ideal for scenarios requiring fast LLM inference, such as chatbots, real-time language processing, and automated content generation.

Quick Start

Use the tensorrt-llm skill to start a local server and generate completions from the provided prompts.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for low latency and high throughput on NVIDIA GPUs?▼

To optimize LLM inference for low latency and high throughput on NVIDIA GPUs, you can use TensorRT to build highly optimized engines. This Skill supports batch processing and quantization, delivering significant speedups compared to standard PyTorch implementations for production environments.

Does TensorRT support FP8 and INT4 quantization for LLM inference?▼

Yes, TensorRT supports both FP8 and INT4 quantization for LLM inference. Applying these quantization techniques accelerates inference speed and reduces memory usage, enabling more efficient deployment of large language models on supported NVIDIA hardware.

What is the best way to scale LLM inference across multiple GPUs for production serving?▼

The best way to scale LLM inference across multiple GPUs is using TensorRT's multi-GPU support. It optimizes model execution across available hardware, efficiently processing multiple simultaneous queries to meet high production demand while maintaining low latency.

Can I use TensorRT for batch inference to process multiple LLM queries simultaneously?▼

Yes, you can use TensorRT for batch inference to process multiple LLM queries simultaneously. It enables efficient batch processing of concurrent requests, which significantly enhances overall inference throughput and optimizes resource utilization on NVIDIA GPUs.

How do I start a local server for LLM inference using TensorRT?▼

To start a local server for LLM inference using TensorRT, use this Skill to initialize the environment and generate completions from provided prompts. This setup leverages NVIDIA TensorRT optimizations to deliver high-throughput and low-latency language model responses locally.

When should I use TensorRT instead of standard PyTorch for LLM inference?▼

You should use TensorRT instead of standard PyTorch for LLM inference when deploying production environments that demand maximum performance. TensorRT provides significant speedups, reduced latency, and advanced memory optimization through quantization that standard PyTorch implementations lack.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
hung-phanhung-phan
1

triton-inference-server

Serve ML models with dynamic batching and concurrent execution via Triton Inference Server.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced