add-memory-prints

What problem does it solve?

Add detailed memory profiling instrumentation throughout the pithtrain training framework. This enables visibility into distributed setup, model creation, checkpoint loading, pipeline scheduling, per-layer activations, saved tensor profiling, expert MLP internals, and memory snapshot dumps to diagnose memory-related bottlenecks.

Core Features & Use Cases

• End-to-end memory profiling across distributed MoE training, including setup, pipeline steps, and per-layer execution.
• Memory snapshots and dumps for activations, weights, and caches to ground memory budgets and identify leaks.
• Targeted per-layer insight with optional detail layers to focus on specific network components during debugging.
• Use Case: You suspect a memory spike during the gating or expert MLP stage and want precise footprints to guide optimization.

Quick Start

Run the memory instrumentation on your training script to enable detailed memory profiling during a run.