skills.rest
projectedanx
projectedanxCOMMUNITY
Explore all skills by this author →
𝕏@ALifeInArtifyAIGitHub

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT-LLM for multi-GPU deployment.

1|Updated May 12, 2026
Search Tags:#gpu#throughput#quantization#inference#nvidia#tensorrt-llm
One-click install
npx skills add https://github.com/projectedanx/hermes-agent --skill tensorrt-llm-projectedanx
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/projectedanx/hermes-agent/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/projectedanx/hermes-agent --skill tensorrt-llm-projectedanx

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This skill addresses the high latency and low throughput bottlenecks inherent in standard LLM inference, enabling production-grade performance on NVIDIA hardware.

Core Features & Use Cases

  • Inference Optimization: Leverages TensorRT-LLM to achieve up to 100x faster inference compared to standard PyTorch implementations.
  • Advanced Scaling: Supports complex multi-GPU and multi-node deployments using Tensor, Pipeline, and Expert parallelism.
  • Production Readiness: Provides built-in support for FP8/INT4 quantization, in-flight batching, and OpenAI-compatible API serving.

Quick Start

Use the tensorrt-llm skill to deploy the Llama-3-70B model across four GPUs with FP8 quantization enabled for maximum throughput.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I maximize LLM inference throughput on NVIDIA GPUs?▼

Maximize LLM inference throughput on NVIDIA GPUs by leveraging TensorRT-LLM for low-latency response times, in-flight batching, and paged KV cache management. This approach achieves up to 100x faster inference compared to standard PyTorch implementations.

Does TensorRT-LLM support multi-GPU scaling for large language models?▼

Yes, TensorRT-LLM supports multi-GPU scaling for large language models using Tensor, Pipeline, and Expert parallelism. This enables complex multi-node deployments for production environments requiring high throughput.

How do I deploy Llama-3-70B across four GPUs with FP8 quantization?▼

Deploy Llama-3-70B across four GPUs with FP8 quantization by using TensorRT-LLM for maximum throughput. The framework provides built-in support for FP8 and INT4 quantization to optimize production performance.

What is in-flight batching and paged KV cache management in LLM inference?▼

In-flight batching and paged KV cache management are LLM inference optimization techniques supported by TensorRT-LLM. They dynamically manage memory and request scheduling to reduce latency bottlenecks during high-throughput production serving.

Can I serve models with an OpenAI-compatible API using TensorRT-LLM?▼

Yes, you can serve models with an OpenAI-compatible API using TensorRT-LLM. It provides built-in production readiness for API serving alongside CUDA graph optimization and quantization features.

What is the best way to optimize PyTorch LLM models for production?▼

The best way to optimize PyTorch LLM models for production is converting them using TensorRT-LLM. It bypasses standard PyTorch bottlenecks by applying CUDA graph optimization, FP8 quantization, and multi-GPU parallelism.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
189

deepstream-import-vision-model

Automate object detection model integration into NVIDIA DeepStream pipelines.

Official
Advanced
jayll1303jayll1303
17

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

Community
Advanced
hung-phanhung-phan
1

triton-inference-server

Serve ML models with dynamic batching and concurrent execution via Triton Inference Server.

Community
Advanced
dawsonblockdawsonblock

serving-llms-vllm

Serve OpenAI-compatible LLM endpoints with vLLM and PagedAttention.

Community
Advanced