Detect WandB metric anomalies during active model training sessions.
npx skills add https://github.com/KwongFuk/codex-skills --skill training-check-kwongfuk
Please help me install this Agent Skill. Skill: training-check Source: https://github.com/KwongFuk/codex-skills/tree/main/global/training-check Command: npx skills add https://github.com/KwongFuk/codex-skills --skill training-check-kwongfuk
Periodically read WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks.
Create a recurring job that runs /training-check <entity>/<project>/<run_id> every 10 minutes.
High-intent search queries and answers about installing and using this skill.
You can monitor WandB metrics for anomalies by setting up a recurring job that fetches run history and analyzes signals like loss trends, NaN/Inf values, and gradient norms during ML training. This enables automated health checks at adjustable intervals.
Yes, you can detect NaN values or loss divergence automatically by running recurring health checks on active training sessions. The system analyzes WandB metrics and alerts you when signals degrade, preventing wasted GPU hours on broken runs.
To run automated health checks when the WandB API is unavailable, the system falls back to reading training logs via SSH. This ensures you can still monitor metrics and detect anomalies even without direct API access.
You can monitor several ML training signals for spikes and anomalies, including loss trend, eval metrics, NaN/Inf values, learning rate, and gradient norms. These are fetched via the WandB API to detect early problems during active model training.
You should check WandB metrics every 10 minutes to catch training problems early. By creating a recurring job at adjustable intervals, you can monitor active runs for loss divergence or idle GPUs and avoid wasting compute resources.
Yes, training-check works with SSH log reads for monitoring ML training. If the WandB API is unavailable, it automatically falls back to reading training logs via SSH to detect metric anomalies and signal degradation.
satsuki-64Monitor WandB metrics during training to detect anomalies and decide whether to continue or halt.
Mang30Detect training quality failures in Weights & Biases runs.
hve4638Monitors WandB metrics during ML training to detect anomalies and escalate uncertainty.
THUFanZdMonitor WandB metrics to detect NaN, divergence, or idle GPUs.
