pytorch-lightning

Move your model into a LightningModule subclass, define training_step and configure_optimizers methods, then pass the module and DataLoader to Trainer.fit. The Trainer handles device placement, backward passes, and optimizer steps automatically.

Lightning provides a structured framework with built-in callbacks, logging, and checkpointing, while Accelerate makes minimal changes to existing loops with more flexibility. Choose Lightning for standardized team workflows and Accelerate for incremental adoption.

Yes, set num_nodes in the Trainer and use DDP, FSDP, or DeepSpeed strategies. Lightning auto-detects SLURM and Kubernetes environments, requiring only MASTER_ADDR, MASTER_PORT, and NODE_RANK variables for manual setups.

Validation only runs if you pass a validation DataLoader to trainer.fit as the second argument. Calling trainer.fit(model, train_loader) without val_loader skips validation entirely.

Reduce batch size, enable gradient accumulation with accumulate_grad_batches, or switch to bf16 precision to cut memory roughly in half. For very large models, use FSDP with FULL_SHARD or DeepSpeed ZeRO-3 with CPU offload.

Use DDP for models under roughly 1-7B parameters where each GPU can hold a full copy. Switch to FSDP with FULL_SHARD for 7-70B parameter models, and DeepSpeed ZeRO-3 with CPU offload beyond that.