skills.rest
InverterNetwork
InverterNetworkOFFICIAL
Explore all skills by this author →
Website𝕏@inverternetworkGitHub

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with quantization and parallelization.

Updated May 4, 2026
Search Tags:#llm#gpu#quantization#inference#serving#nvidia#tensorrt
One-click install
npx skills add https://github.com/InverterNetwork/hermes-agent --skill tensorrt-llm-inverternetwork
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/InverterNetwork/hermes-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/InverterNetwork/hermes-agent --skill tensorrt-llm-inverternetwork

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This skill addresses the challenge of deploying large language models with high latency and low throughput, providing a production-grade solution for optimizing inference on NVIDIA hardware.

Core Features & Use Cases

  • Performance Optimization: Implements advanced techniques like FP8 quantization, in-flight batching, and paged KV cache to maximize tokens per second.
  • Multi-GPU Scaling: Supports tensor, pipeline, and expert parallelism to run massive models across multiple GPUs and nodes.
  • Use Case: Deploy a Llama 3-70B model on a multi-GPU cluster to achieve 24,000+ tokens per second for real-time, high-concurrency applications.

Quick Start

Use the tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with a batch size of 256 on your available NVIDIA GPUs.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference throughput on NVIDIA GPUs?▼

To optimize LLM inference on NVIDIA GPUs, this skill applies FP8 quantization, in-flight batching, and paged KV cache to maximize tokens per second and reduce generation latency for production serving.

Can I run multi-node LLM inference with tensor and pipeline parallelism?▼

Yes, multi-node LLM inference is supported using tensor, pipeline, and expert parallelism to run massive models like Llama 3-70B across multiple GPUs and nodes for high-throughput serving.

What is the best way to deploy a Llama model for high-concurrency serving?▼

The best way to deploy Llama models for high-concurrency serving is using advanced quantization and parallelization techniques, achieving over 24,000 tokens per second on multi-GPU clusters.

Does TensorRT-LLM require specific dependencies for model quantization?▼

Yes, model quantization and inference optimization require both tensorrt-llm and torch dependencies to implement memory-efficient KV caching and high-throughput generation on NVIDIA hardware.

How does paged KV cache improve low-latency LLM generation?▼

Paged KV cache improves low-latency LLM generation by optimizing memory efficiency during inference, allowing larger batch sizes and higher throughput without exhausting GPU memory resources.

Related Skills

Explore More Software Engineering Skills→
Quill-AgentQuill-Agent
3

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs using TensorRT-LLM.

Community
Advanced
ovachieverovachiever
52

serving-llms-vllm

Deploy OpenAI-compatible LLM APIs with vLLM for high-throughput inference.

Community
Advanced
lucasfthlucasfth

serving-llms-vllm

Deploys high-throughput LLM serving with PagedAttention and continuous batching for OpenAI-compatible endpoints.

Community
Advanced
granietgraniet
247

vllm

Launches vLLM-based OpenAI-compatible endpoints with PagedAttention and continuous batching.

Community
Advanced