Distributed Memory Scaling (FSDP / DeepSpeed ZeRO)

What problem does it solve?

The skill provides a unified, config-driven approach to running distributed training across multiple backends (DDP, FSDP, and DeepSpeed ZeRO) with portable weights and cross-strategy checkpointing.

Core Features & Use Cases

• StrategyRouter enables same training scripts to switch between DDP, FSDP, and DeepSpeed ZeRO purely via configuration.
• Lightweight wrappers and utilities handle portable full-state dict export, sharded checkpointing, and deterministic DeepSpeed JSON config generation.
• Cross-strategy portability and cross-world-size resume workflows empower scalable experimentation and production training pipelines.
• A scaling benchmark scaffold measures single-GPU vs multi-GPU performance, with a metrics schema compatible with performance regression gates.

Quick Start

Configure StrategyRouter with your chosen strategy, wrap the model, create the optimizer, and run training using a single, config-driven code path.