skills.rest
GarrettRoi
GarrettRoiCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimize LLM inference with TensorRT-LLM on NVIDIA GPUs.

Updated Mar 12, 2026
Search Tags:#optimization#llm#gpu#inference#serving#nvidia#tensorrt
One-click install
npx skills add https://github.com/GarrettRoi/open-manus --skill tensorrt-llm-garrettroi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/GarrettRoi/open-manus/tree/main/skills/mlops/inference/tensorrt-llm
Command: npx skills add https://github.com/GarrettRoi/open-manus --skill tensorrt-llm-garrettroi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This Skill optimizes Large Language Model (LLM) inference to achieve significantly higher throughput and lower latency, making it suitable for production deployment on NVIDIA hardware.

Core Features & Use Cases

  • High-Performance Inference: Leverages NVIDIA TensorRT-LLM for 10-100x faster inference compared to standard frameworks like PyTorch.
  • Advanced Optimizations: Supports quantization (FP8, INT4), in-flight batching, Paged KV cache, and multi-GPU scaling.
  • Use Case: Deploying a large language model like Llama 3-70B in a production environment where real-time responses and high user concurrency are critical.

Quick Start

Use the tensorrt-llm skill to serve the meta-llama/Meta-Llama-3-8B model with tensor parallelism across 4 GPUs.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I accelerate LLM inference for production deployment on NVIDIA GPUs?▼

To accelerate LLM inference on NVIDIA GPUs, use TensorRT-LLM for production deployment, achieving high throughput and low latency 10-100x faster than standard PyTorch frameworks.

Does TensorRT-LLM support quantization and in-flight batching for large language models?▼

Yes, TensorRT-LLM supports advanced optimizations including FP8 and INT4 quantization, in-flight batching, and Paged KV cache to maximize inference efficiency for large language models.

Can I use multi-GPU tensor parallelism to serve Llama 3 models with TensorRT-LLM?▼

Yes, you can serve models like Llama 3 using TensorRT-LLM with multi-GPU and multi-node parallelism, applying tensor parallelism across multiple GPUs to handle large-scale inference.

What is the best way to optimize Qwen and Mixtral models for low latency serving?▼

The best way to optimize Qwen and Mixtral models for low latency serving is using TensorRT-LLM, which provides specialized inference optimizations including Paged KV cache and multi-GPU scaling.

Do I need PyTorch installed to run TensorRT-LLM optimizations?▼

Yes, PyTorch is a required dependency alongside TensorRT-LLM to build and execute the optimized inference engines for your large language models.

Why does standard PyTorch inference struggle with high user concurrency in production?▼

Standard PyTorch inference lacks advanced features like in-flight batching and Paged KV cache, resulting in lower throughput and higher latency compared to TensorRT-LLM during high user concurrency.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
automatedigitalautomatedigital
1

serving-llms-vllm

Serve large language models with vLLM's optimized inference engine.

Official
Advanced
ancolemanancoleman
503

model-serving

Deploy LLM and ML models for production inference with vLLM, TensorRT-LLM, Ollama, and BentoML.

Community
Advanced
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced