pytorch-lightning

Move your model into a LightningModule subclass, define training_step and configure_optimizers, then call trainer.fit with your DataLoader. Lightning handles device placement, backward passes, and optimizer steps automatically.

Lightning provides a structured framework with built-in callbacks, logging, and checkpointing, while Accelerate makes minimal changes to existing loops with more flexibility. Choose Lightning for standardized team codebases and Accelerate for incremental adoption.

Yes, set num_nodes in the Trainer and configure MASTER_ADDR, MASTER_PORT, and NODE_RANK environment variables on each machine. Lightning also auto-detects SLURM cluster environments for job scheduling.

Validation only runs when you pass a validation DataLoader to trainer.fit as the second argument. Calling trainer.fit(model, train_loader) without val_loader skips validation entirely.

Reduce batch size, enable gradient accumulation with accumulate_grad_batches, or switch to bf16 precision to cut memory usage roughly in half. For very large models, use FSDP with cpu_offload enabled.

Use DDP for models under roughly 7 billion parameters where each GPU can hold a full copy. Switch to FSDP with FULL_SHARD for 7-70B parameter models, and DeepSpeed ZeRO-3 with CPU offload beyond that.