Precision + Numerics Stabilizer (bf16/fp16 Done Right)

Preventing NaNs during mixed-precision training requires a numerics pipeline with AMP autocast and GradScaler. This stabilizer automatically detects non-finite values, manages gradient scaling, and logs failure snapshots to ensure safe unattended fp16 and bf16 workflows.

Configuring GradScaler for fp16 training loops involves automatic scale management and safe unscale/clip ordering. This Skill integrates GradScaler directly into the PyTorch workflow to dynamically adjust scaling factors and prevent gradient underflow or overflow.

PyTorch autocast safely supports flexible bf16, fp16, and fp32 precision modes when paired with a numerics sentinel. This Skill enforces master-weights-in-fp32 invariants and applies autocast constraints to maintain stability across CPUs and GPUs.

Capturing reproducible snapshots when a training run crashes involves saving diagnostic data with atomic writes. This Skill generates 7-file failure snapshots containing repro scripts and numerics logs to investigate NaNs, Inf values, and gradient anomalies.

Loss spikes or NaN outputs during AMP autocast often stem from gradient overflow or insufficient scaling. This Skill monitors gradient norms, logits, and activation checks to detect non-finite values and automatically abort or snapshot the failure for debugging.

Running unattended mixed-precision training safely requires continuous numerics instrumentation to detect non-finite values. This Skill provides a configurable precision pipeline with automatic NaN detection and gradient norm monitoring to prevent silent failures.