skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

ad-sharding-ir-port

Adds sharding-aware IR hints to TensorRT-LLM AutoDeploy custom model files in place.

14.5k|2.7k|Updated Aug 16, 2023
Search Tags:#llm inference#autodeploy#model porting#tensorRT-llm#Tensor Parallelism#Sharding#Multi-GPU
One-click install
npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill ad-sharding-ir-port
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ad-sharding-ir-port
Source: https://github.com/NVIDIA/TensorRT-LLM/tree/main/.claude/skills/ad-sharding-ir-port
Command: npx skills add https://github.com/NVIDIA/TensorRT-LLM --skill ad-sharding-ir-port

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Porting an AutoDeploy custom model to tensor/expert parallelism requires inserting precise sharding hints (op substitutions, tp_mode kwargs, all_reduce insertions) into the modeling file without altering its numerical semantics or module structure, which is error-prone when done ad hoc.

Core Features & Use Cases

  • Mechanical hint insertion: Substitutes nn.Linear, view, and split calls with torch.ops.auto_deploy equivalents and adds tp_mode, layer_type, output_sizes, and tp_scaled_dim kwargs under a strict allowlist/forbidden-change contract.
  • Layer-specific patterns: Provides sharding recipes for MHA, MLA, SwiGLU MLP, MoE, Mamba/SSM, and GatedDeltaNet layers, including single all_reduce merge-point rules.
  • Mandatory validation: Enforces an end-to-end multi-GPU run plus a four-configuration sharding equivalence test (tp-only, ep-only, tep, attn-dp) with parsed rel_rmse below tolerance, and a git-diff self-audit classifying every hunk.
  • Use Case: When onboarding a new LLM such as a Qwen or DeepSeek variant into TensorRT-LLM AutoDeploy, use this Skill to port its modeling_*.py file to the sharding IR path and verify correctness across parallelism configurations.

Quick Start

Add sharding IR hints to tensorrt_llm/torch/auto_deploy/models/custom/modeling<name>.py and validate it with the sharding equivalence test matrix.

Frequently Asked Questions about ad-sharding-ir-port

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add tensor parallelism sharding hints to an AutoDeploy custom model?▼

Replace nn.Linear calls with torch.ops.auto_deploy.torch_linear_simple using tp_mode colwise for opening projections and rowwise for closing ones, insert all_reduce after rowwise projections, and use auto_deploy view and split_with_sizes ops where dimensions scale with TP.

How do I validate a sharding IR port in TensorRT-LLM AutoDeploy?▼

Run the sharding equivalence test test_sharding_num_correctness.py across all four configurations (tp-only, ep-only, tep, attn-dp) and confirm each cell's rel_rmse is strictly below the logged tolerance. Also complete one end-to-end multi-GPU run with build_and_run_ad.py.

Does the sharding IR path require a separate _ir.py model file?▼

No. Sharding hints are added directly to the canonical modeling_*.py file in place, and the legacy pattern of maintaining parallel modeling_*_ir.py files is deprecated. The apply_sharding_hints transform auto-detects all_reduce markers in the exported FX graph.

Why does the sharding equivalence test fail with high rel_rmse?▼

A rel_rmse above tolerance indicates a sharding-hint bug such as a missing all_reduce, a wrong tp_mode, a view without tp_scaled_dim, or split sizes that do not scale with TP. Re-check the layer-specific patterns and iterate on the hints rather than raising the tolerance.

What changes are forbidden when porting a model to sharding IR?▼

You must not replace torch.ops.trtllm.* ops with vanilla PyTorch, change forward input contracts, alter module hierarchy or state_dict keys, modify numerical semantics like dtype or mask fill values, rename methods, or remove existing code. Any change outside the allowlist requires explicit human approval.

Related Skills

Explore More Software Engineering Skills→
yo-stevenyo-steven

ad-model-onboard

Translate HuggingFace models into prefill-only AutoDeploy custom models with validation.

Community
Advanced
attentiondotnetattentiondotnet

tensorrt-llm

Optimize LLM inference on NVIDIA GPUs with TensorRT-LLM.

Community
Advanced
NVIDIANVIDIA
14.3k

ad-model-onboard

Translate HuggingFace models into prefill-only AutoDeploy custom models with equivalence tests.

Official
Advanced
pwretmopwretmo

review-thread-resolution

Resolve GitHub PR review threads via gh api graphql after code changes.

Community
Intermediate