Analyze WandB training metrics to detect early failure signals.
npx skills add https://github.com/Shallow-W/llm-wiki --skill training-check-shallow-w
Please help me install this Agent Skill. Skill: training-check Source: https://github.com/Shallow-W/llm-wiki/tree/main/.claude/skills/training-check Command: npx skills add https://github.com/Shallow-W/llm-wiki --skill training-check-shallow-w
It prevents wasted GPU time by detecting early signs of training failure such as NaNs, loss divergence, or non-improving metrics while a run is still in progress.
Use training-check for a WandB run by passing the run path in the form entity/project/run_id so the skill can evaluate loss trend and eval metrics on a periodic interval.
High-intent search queries and answers about installing and using this skill.
To automatically detect loss divergence or NaNs, you can periodically monitor training metrics to catch early failure signals like non-improving eval performance or spikes while the run is still in progress.
Yes, you can check WandB metrics by passing the run path as entity/project/run_id, and the system will periodically re-check at an interval that expands up to a cap when metrics remain healthy.
If WandB is unreachable, metric health checks fall back to guiding direct log reading via SSH to inspect training trends, preventing false assumptions about run quality.
Training monitoring works by analyzing both loss trends and learning rate behavior alongside eval performance and NaN/Inf events, classifying these signals into clearly good, clearly bad, or ambiguous states.
You should use early failure detection for running deep learning training jobs to prevent wasted GPU time by stopping compute when metrics indicate non-improving states or loss divergence.
lightrain-aMonitor WandB training metrics to detect loss divergence, NaN/Inf, and eval degradation.
Albatross679Monitor ML training jobs and auto-restart from the last checkpoint on crash.
Mang30Detect training quality failures in Weights & Biases runs.
THUFanZdMonitor WandB metrics to detect NaN, divergence, or idle GPUs.
