model-infer-fusion

Replaces PyTorch model operators with torch_npu fused operators for Ascend NPU inference.

Updated Sep 15, 2026
One-click install
npx skills add https://github.com/WangWindow/CANN-BatchMatMulMaxsum --skill model-infer-fusion-wangwindow
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: model-infer-fusion
Source: https://github.com/WangWindow/CANN-BatchMatMulMaxsum/tree/main/.agents/skills/model-infer-fusion
Command: npx skills add https://github.com/WangWindow/CANN-BatchMatMulMaxsum --skill model-infer-fusion-wangwindow

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch_npu, and includes scripts (resource) and references (resource) components.

What problem does it solve? Manually identifying which parts of a PyTorch model can be replaced with torch_npu fused operators on Ascend NPU is error-prone: developers must decompose Attention, MoE, FFN, and Norm modules, match them against a large operator catalog, and verify API signatures, dtype, shape, layout, and cache constraints before replacing code. This Skill provides a structured analyze-match-replace-validate workflow grounded in local torch_npu docstring queries and repository reference implementations. ## Core Features & Use Cases - Module decomposition and matching: Breaks models into replaceable sub-chains (RoPE, KV Cache write, Flash Attention, MoE gating/routing/dispatch, Residual+Norm) and matches each against repository reference implementations for GQA, MLA Absorb, and MLA+Indexer paths. - API verification tooling: Ships a query script that reads torch_npu's built-in _op_plugin_docs.py to show, search, and list operator documentation, with fallback entries and an offline operator catalog. - Use Case: Given a GQA model whose Prefill uses current q/k/v and Decode reads from a PA KV Cache, the Skill guides separate analysis of each stage, verifies npu_fused_infer_attention_score constraints via local docstrings, then replaces and validates accuracy and performance module by module. ## Quick Start Analyze my PyTorch model code and tell me which modules can be replaced with torch_npu fused operators, with candidate APIs and a validation plan.

Frequently Asked Questions about model-infer-fusion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I replace PyTorch operators with torch_npu fused operators?

Follow a four-step workflow: decompose the model into replaceable sub-chains, match each module against repository reference implementations, verify candidate APIs via local torch_npu docstring queries, then replace and validate module by module with accuracy and performance comparisons.

How to check torch_npu operator API signatures before replacing code?

Use the bundled query script to run show, search, or list commands against torch_npu's built-in _op_plugin_docs.py docstrings. This avoids relying on memory and confirms parameters, dtype, shape, and layout constraints for your installed version.

Which model modules can torch_npu fused operators cover?

Covered modules include Attention sub-chains (RoPE, KV Cache write, Flash Attention for GQA and MLA paths), MoE gating, routing, dispatch and combine, Dense FFN, and Norm plus residual fusions. Prefill and Decode branches are analyzed separately.

Can I reuse a torch_npu call from another model implementation directly?

No. Repository reference implementations serve only as candidate blueprints. You must still verify the operator's signature, shape, dtype, layout, cache, and metadata constraints against official documentation for your current torch_npu version before adapting it.

What are the limitations of torch_npu fused operator replacement?

The Skill only identifies and replaces existing torch_npu fused operators; it does not design quantization algorithms or develop custom kernels. Modules blocked by hard constraints or external tensor consumers are documented as inapplicable or new operator requirements.

What should I do when a fused operator replacement fails accuracy validation?

Roll back the module's changes and record the accuracy results, error evidence, and attempted fixes. Then evaluate alternative existing operators, and only file a new fused operator requirement when no current API fits and clear fusion benefit exists.