comm-analysis

What problem does it solve?

This Skill analyzes communication primitives in TPU pretraining profiles, providing insights into communication bandwidth utilization and overlap with computation. It helps identify performance bottlenecks and optimize TPU utilization.

Core Features & Use Cases

• Communication Analysis: List, attribute, and analyze communication primitives (async/sync, TC/SC, P2P) with rich attributes like bandwidth, latency, and overlap.
• Per-axis Bandwidth Utilization: Compute and compare NCCL bus bandwidth and peak ICI link bandwidth for each axis, with utilization percentage.
• Compute/Comm Overlap: Calculate and report compute-to-communication overlap for each step.
• Critical-path Attribution: Identify and report the critical-path exposure for each communication primitive.
• Use Case: Analyze a TPU pretraining profile to identify high-bandwidth communication primitives and optimize data placement or collective communication patterns for better performance.

Quick Start

Use the comm-analysis skill to analyze the communication primitives in the profile located at /tmp/tensorboard/tensorboard/plugins/profile/dp8_fsdp128.