skills.rest
cxnaive
cxnaiveCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize large language model inference with NVIDIA TensorRT-LLM.

Updated Jun 17, 2026
Search Tags:#optimization#llm#gpu#inference#serving#nvidia#tensorrt-llm
One-click install
npx skills add https://github.com/cxnaive/hermes-agent-llbot --skill tensorrt-llm-cxnaive
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/cxnaive/hermes-agent-llbot/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/cxnaive/hermes-agent-llbot --skill tensorrt-llm-cxnaive

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This skill addresses the challenge of deploying large language models in production environments where high throughput, low latency, and efficient GPU memory utilization are critical requirements.

Core Features & Use Cases

  • Inference Optimization: Leverages techniques like FP8/INT4 quantization, in-flight batching, and paged KV cache to maximize performance on NVIDIA hardware.
  • Multi-GPU Scaling: Supports tensor, pipeline, and expert parallelism to run massive models across multiple GPUs and nodes.
  • Use Case: Deploy a Llama 3-70B model on a multi-GPU cluster to achieve 24,000+ tokens per second for high-traffic production applications.

Quick Start

Use the tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with a batch size of 256 on your local GPU.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference for high-throughput production environments on NVIDIA GPUs?▼

You can optimize LLM inference by leveraging NVIDIA TensorRT-LLM with techniques like FP8/INT4 quantization, in-flight batching, and paged KV cache to maximize throughput and minimize latency on NVIDIA hardware.

What's the best way to run large models across multiple GPUs and nodes?▼

Running large models across multiple GPUs requires tensor parallelism, pipeline parallelism, and expert parallelism strategies provided by TensorRT-LLM to efficiently distribute workloads and scale massive models across multi-GPU clusters.

Does TensorRT-LLM support FP8 and INT4 quantization formats?▼

Yes, TensorRT-LLM supports both FP8 and INT4 quantization formats to reduce memory footprint and accelerate inference speeds while maintaining model accuracy on compatible NVIDIA GPUs.

How do I deploy a Llama 3 model with reduced latency for high-traffic applications?▼

Deploying a Llama 3 model for high-traffic applications involves using TensorRT-LLM's in-flight batching, paged KV cache, and speculative decoding features to achieve thousands of tokens per second with reduced latency.

Can I use tensor parallelism and pipeline parallelism together for LLM serving?▼

Yes, you can combine tensor parallelism and pipeline parallelism in TensorRT-LLM to run massive models across multiple GPUs and nodes, enabling efficient model serving for demanding production environments.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
masermediagroup-stackmasermediagroup-stack
1

model-serving

Deploy and manage LLM and ML model-serving stacks with optimized backends.

Community
Advanced
kngender5kngender5

colab-advanced-inference-optimization

Optimize inference speed and VRAM efficiency for Colab models using FlashAttention-2/3.

Community
Advanced