pytorch-patterns

Use torch.amp.GradScaler with torch.amp.autocast to run forward passes in mixed precision, then scale the loss before backward and step the optimizer through the scaler. This typically doubles training speed on compatible GPUs while reducing memory usage.

Set seeds for torch, torch.cuda, numpy, and random, then enable torch.backends.cudnn.deterministic and disable cudnn.benchmark. This ensures identical results across runs, though it may slightly reduce training speed.

This usually happens when model.eval() is not called before validation, leaving dropout active and BatchNorm using batch statistics. Always call model.eval() and wrap inference in torch.no_grad() before evaluating.

Yes, set num_workers greater than zero to load batches in parallel processes. Combine with pin_memory=True for faster CPU-to-GPU transfer and persistent_workers=True to keep workers alive between epochs.

Save a dictionary containing epoch, model_state_dict, optimizer_state_dict, and loss with torch.save. To resume, load with weights_only=True, restore both state dicts, and continue from the saved epoch.

Use torch.utils.checkpoint when training large models that exceed GPU memory, as it recomputes activations during backward instead of storing them. It trades roughly 30% more compute for significant memory savings.