skills.rest
JamesFincher
JamesFincherCOMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Optimizes LLM inference serving on NVIDIA GPUs using TensorRT-LLM with quantization and parallelism options.

Updated May 4, 2026
Search Tags:#model serving#llm inference#int4#multi-gpu#tensorRT-llm#fp8 quantization#in-flight batching
One-click install
npx skills add https://github.com/JamesFincher/gengar --skill tensorrt-llm-jamesfincher
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/JamesFincher/gengar/tree/main/optional-skills/mlops/tensorrt-llm
Command: npx skills add https://github.com/JamesFincher/gengar --skill tensorrt-llm-jamesfincher

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires tensorrt-llm, torch, and includes references (resource) components.

What problem does it solve?

This Skill helps you deploy large language models with dramatically lower latency and higher throughput by optimizing inference using NVIDIA TensorRT-LLM on production NVIDIA GPUs.

Core Features & Use Cases

  • High-throughput inference serving: Build an OpenAI-compatible server with dynamic batching to maximize tokens/sec for real workloads like chat and API traffic.
  • Performance and latency optimization: Enable TensorRT-LLM features such as in-flight batching, paged KV cache, CUDA graphs, and Flash Attention.
  • Production deployment across scales: Use tensor/pipeline/expert parallelism and quantization (FP8/INT4) to scale from single-GPU to multi-GPU/multi-node.

Quick Start

Start the OpenAI-compatible chat server for a Llama 3 model by running the tensorrt-llm serve command with your desired tensor-parallel size and FP8 configuration.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize LLM inference latency and throughput on NVIDIA GPUs?▼

Optimize LLM inference latency and throughput on NVIDIA GPUs by configuring TensorRT-LLM with in-flight batching, paged KV cache, CUDA graphs, and Flash Attention to maximize tokens per second for production traffic.

How do I set up an OpenAI-compatible API server for high-concurrency chat workloads?▼

Set up an OpenAI-compatible API server for high-concurrency chat workloads by running the TensorRT-LLM serve command with dynamic batching to handle real-time API traffic efficiently.

Can I use FP8 or INT4 quantization to scale multi-GPU LLM inference?▼

Use FP8 or INT4 quantization to scale multi-GPU LLM inference by applying tensor, pipeline, or expert parallelism strategies, enabling deployment expansion from single-GPU to multi-node configurations.

Do I need specific CUDA and TensorRT versions to run TensorRT-LLM?▼

Specific CUDA and TensorRT versions are required to run TensorRT-LLM, alongside correct server configuration for batching, KV cache behavior, and distributed sharding to ensure proper execution.

What is the best way to serve Llama models faster with TensorRT?▼

Serve Llama models faster with TensorRT by starting the serve command with your desired tensor-parallel size and FP8 configuration, optimizing inference for dramatically lower latency on NVIDIA GPUs.

Related Skills

Explore More Software Engineering Skills→
ovachieverovachiever
52

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
john-data-chenjohn-data-chen
2

serving-llms-vllm

Configure vLLM servers for OpenAI-compatible endpoints with Docker or Kubernetes.

Community
Advanced
nishide-devnishide-dev

ml-model-export

Export trained PyTorch models to ONNX, TorchScript, and TensorRT formats.

Community
Advanced
twjarviszyra-webtwjarviszyra-web
3

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced