skills.rest
jayll1303
jayll1303COMMUNITY
Explore all skills by this author →
GitHub

tensorrt-llm

Convert HuggingFace checkpoints into optimized TensorRT-LLM engines with quantization.

17|3|Updated Apr 3, 2026
Search Tags:#quantization#triton#fp8#tensorrt#tensorrt-llm#multi-gpu#trtllm-build
One-click install
npx skills add https://github.com/jayll1303/AIEKit --skill tensorrt-llm-jayll1303
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tensorrt-llm
Source: https://github.com/jayll1303/AIEKit/tree/main/.kiro/skills/tensorrt-llm
Command: npx skills add https://github.com/jayll1303/AIEKit --skill tensorrt-llm-jayll1303

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Reduce the time, complexity, and risk of deploying high-throughput LLM inference on NVIDIA GPUs by providing a clear workflow for converting HuggingFace checkpoints, building optimized TensorRT-LLM engines, and configuring production serving options.

Core Features & Use Cases

  • Convert HuggingFace model checkpoints into TensorRT-LLM checkpoint format suitable for engine building.
  • Build optimized TensorRT engines with support for FP8, INT4, and INT8 quantization, GEMM and attention kernel fusion, and multi-profile builds.
  • Configure in-flight batching, paged KV cache, and other runtime tuning parameters to maximize tokens/sec throughput.
  • Support multi-GPU tensor parallelism and pipeline parallelism for large-model builds and Triton backend integration for production serving.
  • Use case: convert a Llama-3 checkpoint, apply FP8 quantization on Hopper GPUs, build rank*.engine files, and deploy the model behind Triton for scalable streaming inference.

Quick Start

Use the tensorrt-llm skill to convert a HuggingFace checkpoint, build an optimized TensorRT engine with the desired quantization, and prepare the engine for Triton serving.

Frequently Asked Questions about tensorrt-llm

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a HuggingFace checkpoint into a TensorRT engine for LLM inference?▼

To convert a HuggingFace checkpoint for LLM inference, you transform it into a TensorRT-LLM checkpoint format and use the trtllm-build command to compile an optimized engine with your desired quantization settings.

What GPU architecture is required for FP8 quantization in TensorRT-LLM?▼

FP8 quantization in TensorRT-LLM requires compatible NVIDIA GPU architectures, specifically Hopper or Ada. You also need sufficient VRAM for the build-time overhead and compatible CUDA and TensorRT runtimes.

Can I deploy TensorRT-LLM engines across multiple GPUs for large models?▼

Yes, TensorRT-LLM supports multi-GPU tensor parallelism and pipeline parallelism for large-model builds. This allows you to distribute memory and computation across multiple NVIDIA GPUs to maximize throughput.

How does in-flight batching and paged KV cache improve LLM throughput?▼

In-flight batching and paged KV cache are runtime tuning parameters that maximize tokens per second throughput. They enable continuous dynamic request insertion and efficient memory management during production LLM inference.

What is the best way to serve a quantized TensorRT-LLM model in production?▼

The best way to serve a quantized TensorRT-LLM model in production is by deploying the built rank engine files behind Triton. This backend integration enables scalable, high-throughput streaming inference.

Related Skills

Explore More Software Engineering Skills→
anilcan-karaanilcan-kara

tensorrt-llm

Optimize LLM inference with NVIDIA TensorRT for high throughput and low latency.

Community
Advanced
ricablericable

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and parallelism.

Community
Advanced
hung-phanhung-phan
1

triton-inference-server

Serve ML models with dynamic batching and concurrent execution via Triton Inference Server.

Community
Advanced
Orchestra-ResearchOrchestra-Research
11.5k

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs using TensorRT-LLM with quantization and batching.

Official
Advanced