shmem-ops-performance-optim

Optimizes SHMEM operator performance through five iterative mechanism-tuning rounds with delta comparisons.

Updated Sep 15, 2026
One-click install
npx skills add https://github.com/WangWindow/CANN-BatchMatMulMaxsum --skill shmem-ops-performance-optim-wangwindow
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: shmem-ops-performance-optim
Source: https://github.com/WangWindow/CANN-BatchMatMulMaxsum/tree/main/.agents/skills/shmem-ops-performance-optim
Command: npx skills add https://github.com/WangWindow/CANN-BatchMatMulMaxsum --skill shmem-ops-performance-optim-wangwindow

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? SHMEM communication and fused compute-communication operators on Ascend NPU often underperform their HCCL baselines, and manual bottleneck analysis across communication bandwidth, synchronization, core utilization, memory efficiency, and pipeline depth is slow and error-prone. This Skill runs a structured five-round optimization loop that locates bottlenecks, delegates code changes, and verifies each step with correctness and performance re-measurement. ## Core Features & Use Cases - Bottleneck Diagnosis: Reads performance reports and design docs to classify bottlenecks across five categories (communication bandwidth, synchronization overhead, core utilization, memory efficiency, pipeline depth), including algorithm-topology mismatch checks. - Delegated Optimization Rounds: Outputs structured change proposals and delegates design changes to shmem-ops-design, code changes to shmem-ops-code-gen, builds to shmem-ops-compile-debug, and correctness re-tests to shmem-ops-correctness-eval. - Round-by-Round Verification: Each round re-runs performance evaluation via shmem-ops-performance-eval, records step-level metrics (kernel_bus_bandwidth_GBps, e2e/kernel latency, compute utilization), and makes keep/revert decisions with delta-percent comparisons against baseline. - Use Case: After a SHMEM allgather or matmul_allreduce operator passes correctness but reaches only 65% of the HCCL baseline bandwidth, run this Skill to execute five mechanism-optimization rounds (e.g., concurrent peer communication, ping-pong double buffering, barrier-to-signal replacement) and produce a complete performance_report.md. ## Quick Start Ask the agent to run the SHMEM performance optimization loop for the operator that finished Phase 6 performance collection but missed its target, and complete all five optimization rounds with a final performance report.

Frequently Asked Questions about shmem-ops-performance-optim

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize SHMEM operator performance on Ascend NPU?

Run a structured five-round loop: locate bottlenecks from the performance report, propose one mechanism change per round (overlap, double buffering, signal/wait, engine switch), delegate code changes, then re-verify correctness and re-measure kernel_bus_bandwidth_GBps each round.

What is the difference between mechanism optimization and parameter tuning for SHMEM operators?

Mechanism optimization changes code logic or algorithms, such as adding ping-pong double buffers, sender/receiver core splits, or replacing barriers with signal/wait. Parameter tuning only adjusts configuration values like block_dim, chunk_size, or TileShape without changing code structure.

When should I use SDMA versus MTE for SHMEM data transfers?

Use MTE for transfers under 2MB due to lower setup overhead, and SDMA for transfers of 2MB or more within a node where P2P is reachable. For cross-node traffic, RDMA/RoCE is the only available path.

Why is my SHMEM operator far below the HCCL baseline bandwidth?

When performance is under 50% of baseline, first check algorithm-topology mismatch, such as serial per-peer fetches on a full-mesh topology that use only one of several HCCS links. Then examine chunk size, missing overlap, global barriers, and serial cross-PE loops inside a single AIV.

Can the optimization loop stop early once the performance target is met?

No. Reaching the target is not an early-stop condition; once Phase 6.5 is entered, all configured rounds (default five) must complete to fully explore the optimization space. Each round still requires correctness re-testing and performance re-measurement.

What metrics are used to compare optimization rounds?

Round-to-round comparisons must use kernel_bus_bandwidth_GBps as the primary metric, never e2e bandwidth. Step records also capture e2e_latency_us, kernel_latency_us, algo_bandwidth_GBps, and compute_utilization for fused compute-communication operators.