cluster-fast-slow-rank-detector

Diagnosing slow cards in Ascend cluster profiling involves classifying performance bottlenecks into Host delivery, compute, or communication issues using expert rules and scripted comparisons, producing macro conclusions and micro evidence.

Comparing API and operator statistics between fast and slow ranks requires executing scripts like compare_api_stats.py and compare_op_stats.py to identify the top 20 timing differences, pinpointing the exact bottleneck points.

Common root causes for slow cards in multi-rank profiler data include Host delivery bottlenecks, pure compute slowness, and communication delays, which are automatically classified by applying expert rules to the profiling data.

Yes, this slow card diagnosis approach requires the pandas dependency to process the profiling data and mandates executing the scripts within the scripts directory to enforce a deterministic comparison workflow.

The best way to identify performance bottlenecks in multi-rank cluster profiling is applying expert rules to classify slow cards and running comparison scripts to drill down from macro bottlenecks to micro evidence.