What problem does it solve? Training large models on Ascend NPU often crashes with out-of-memory errors, and engineers struggle to determine whether the root cause is an oversized configuration, temporary workspace/HCCL buffer pressure, memory fragmentation, or a gradual memory leak. This Skill provides a systematic workflow to classify the OOM type from logs, statically estimate memory demand, analyze memory snapshots, and produce targeted optimization recommendations. ## Core Features & Use Cases - OOM Classification Workflow: Categorizes failures by source (NPUWorkspaceAllocator, HCCL communication, NPUCachingAllocator) and timing (step 0-2 configuration issue vs. late-step progressive leak) to route the diagnosis correctly. - Static Memory Estimation: Estimates parameter, gradient, optimizer state, activation, and temporary buffer memory against device capacity before collecting any runtime data. - Snapshot Deep Analysis: Ships an analyze_snapshot.py script that parses PyTorch memory snapshot pickle files into six dimensions: allocation overview, semantic category breakdown, peak analysis, timeline/leak detection, fragmentation depth, and top-N large allocations, plus multi-snapshot comparison. - Optimization Decision Matrix: Maps each diagnosis to concrete fixes such as memory waterline ratios, activation checkpointing, swap optimizer, parallelism adjustments, and leak-fix code review checklists. - Use Case: A training run crashes at step 15 with 'NPU out of memory'. The Skill identifies it as a suspected leak, compares snapshots across steps to confirm monotonic growth in communication buffers, and points to missing wait_tensor calls after async redistribute in the parallelization code. ## Quick Start Ask the assistant to diagnose your NPU training OOM by providing the full error log, training configuration, and NPU device memory size.
