skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

trtllm-model-onboard-multimodal

Onboard HuggingFace multimodal models to the TensorRT-LLM PyTorch backend.

14.5k|2.7k|Updated Aug 16, 2023
Search Tags:#inference#huggingface#model onboarding#pyTorch#tensorRT-llm#vision-language model#Multimodal
One-click install
npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill trtllm-model-onboard-multimodal
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trtllm-model-onboard-multimodal
Source: https://github.com/NVIDIA/TensorRT-LLM/tree/main/.claude/skills/trtllm-model-onboard-multimodal
Command: npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill trtllm-model-onboard-multimodal

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Porting a HuggingFace vision-language, audio, or video model to TensorRT-LLM's PyTorch backend requires correctly wiring a multimodal encoder, input processor, weight mapper, and LLM backbone while respecting strict performance contracts (no CPU-GPU syncs, shared-tensor media transport, batched encoder execution). This Skill provides the complete phased workflow, canonical code templates, and audit checks to do it without silent correctness or performance regressions.

Core Features & Use Cases

  • Phased onboarding workflow: Step-by-step phases from resource gathering and coverage survey through model wrapper, input processor, weight mapper, and testing.
  • Module reuse mapping: Tables mapping common layers (Linear, Attention, GatedMLP, RoPE) to their TensorRT-LLM _torch.modules equivalents so quantization, tensor parallelism, and CUDA graphs work automatically.
  • Performance contracts: Four enforceable rules covering zero CPU-GPU syncs in forward, async CPU preprocessing, shared-tensor media transport, and batched encoder execution, each with audit commands.
  • Use Case: Onboard a new Qwen-VL-style vision-language model by creating modeling_<name>.py with a ported encoder, an input processor subclassing BaseMultimodalInputProcessor, a weight mapper, and unit plus accuracy tests.

Quick Start

Onboard the HuggingFace model org/model-name to the TensorRT-LLM PyTorch backend following the multimodal onboarding phases.

Frequently Asked Questions about trtllm-model-onboard-multimodal

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I onboard a HuggingFace vision-language model to TensorRT-LLM?▼

Create a modeling_<name>.py file with a wrapper class registered via @register_auto_model that composes a ported multimodal encoder and an inner LLM resolved through TRT-LLM's AutoModelForCausalLM. Add a BaseMultimodalInputProcessor subclass, a weight mapper if HF prefixes differ, and unit plus accuracy tests.

How do I add multimodal support to the TensorRT-LLM PyTorch backend?▼

Port the encoder to _torch.modules components (Linear, Attention, GatedMLP, RotaryEmbedding), implement call_with_text_prompt in an input processor subclass, and declare multimodal_data_device_paths. Follow the four performance contracts covering sync-free forward, async preprocessing, shared-tensor transport, and batched encoder execution.

Can I reuse transformers library modules in a TensorRT-LLM modeling file?▼

No, new onboardings must port encoder blocks to TRT-LLM modules rather than importing from transformers. HF passthrough loses quantization, tensor parallelism, fused kernels, and CUDA graph support, and creates version coupling. The only existing exception is the legacy Qwen2-VL family.

Does this skill cover TensorRT-LLM AutoDeploy model onboarding?▼

No, this skill covers only the PyTorch backend under tensorrt_llm/_torch used by LLM(backend="pytorch"), trtllm-serve, and trtllm-bench. For the AutoDeploy path under tensorrt_llm/_torch/auto_deploy, use the ad-model-onboard skill instead.

Why does my multimodal model re-run the encoder on every prefill chunk?▼

This happens when the encoder output length does not match the MM placeholder count the input processor placed in prompt_token_ids, or when the encoder returns a list with more than one element. get_multimodal_embeddings then silently skips caching, forcing re-execution each chunk.

What causes CPU-GPU sync stalls in a TensorRT-LLM VLM forward pass?▼

Calling .item(), .tolist(), .cpu(), torch.nonzero, single-arg torch.where, or Python branching on GPU tensor values inside forward forces synchronization and collapses the overlap scheduler. Audit by running one prefill iteration with torch.cuda.set_sync_debug_mode("warn") and confirming zero warnings.

Related Skills

Explore More Software Engineering Skills→
yo-stevenyo-steven

ptq

Quantize pretrained LLMs and VLMs into HuggingFace or TensorRT-LLM checkpoints.

Community
Advanced
gagan114662gagan114662

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM quantization and batching.

Community
Advanced
yo-stevenyo-steven

perf-nsight-systems

Profile CPU, GPU, and communication timelines from .nsys-rep captures.

Community
Intermediate
yo-stevenyo-steven

ad-add-fusion-transformation

Add TensorRT-LLM AutoDeploy fusion transformation passes for graph pattern rewriting.

Community
Advanced