performance-analysis

You can analyze MaxText training performance by running the workflow on completed job artifacts to generate structured metrics, GPU utilization breakdowns, and kernel-level insights using TSDB, TraceLens, and IRLens.

The best way to compare multiple MaxText training jobs is using TSDB-based comparisons within the analysis workflow, which orchestrates single or multi-job diagnostics to pinpoint resource contention and efficiency issues across GPU, host, and network layers.

You can run performance profiling mid-training, not just post-run. The workflow applies guided steps to read results, summarize findings, and validate dashboard availability for both mid-training profiling and post-run diagnostics.

TraceLens works with IRLens in a multi-tool workflow to pinpoint root causes of training efficiency issues. TraceLens generates performance reports while IRLens provides analysis to detect bottlenecks across GPU, host, and network layers.

Low GPU utilization in a MaxText training run can be caused by resource contention across host or network layers. Use the analysis workflow to generate kernel-level insights and actionable metrics that pinpoint the exact bottleneck.