Distributed Scaling

What problem does it solve?

Orchestrates scalable distributed training for BrainAI, enabling efficient multi-GPU/multi-node execution and memory-aware configurations.

Core Features & Use Cases

• Supports DDP and FSDP wrapping, enabling training of models from 1B to 7B parameters across multiple GPUs.
• Provides gradient accumulation, mixed-precision policy, and memory estimation to maximize throughput on constrained hardware.
• Includes multi-node launch tooling, phase-based checkpointing, and per-node data loading strategies for reproducible scaling.

Quick Start

Select a preset (for example for_1b or for_7b) and run your training script with the DistributedLauncher to initialize the distributed environment.