pytorch-lightning

Move your model definition into a LightningModule subclass, put the loss computation in training_step, and return the optimizer from configure_optimizers. Then create a Trainer and call trainer.fit with your DataLoader, removing all manual device and backward-pass code.

Set accelerator='gpu', devices to the GPU count, and strategy='ddp' in the Trainer. Lightning automatically handles process spawning, data distribution with DistributedSampler, and gradient synchronization without code changes.

Lightning provides a fully structured framework with built-in callbacks, checkpointing, and logging, ideal for standardized team workflows. Accelerate makes minimal changes to existing PyTorch loops and offers more flexibility for custom training logic.

Yes, Lightning includes FSDPStrategy with FULL_SHARD for ZeRO-3 equivalent sharding and DeepSpeedStrategy supporting ZeRO stages 2 and 3 with CPU offloading. These strategies target models from 7B to 70B+ parameters.

Validation only runs when you pass a validation DataLoader to trainer.fit as the second argument or via val_dataloaders. Calling trainer.fit(model, train_loader) alone skips validation entirely.

Reduce batch size, enable gradient accumulation with accumulate_grad_batches, or switch to mixed precision with precision='bf16' to cut memory usage roughly in half. For very large models, use FSDP with cpu_offload enabled.