Detect training quality failures in Weights & Biases runs.
npx skills add https://github.com/Mang30/myskills --skill training-check-mang30
Please help me install this Agent Skill. Skill: training-check Source: https://github.com/Mang30/myskills/tree/main/skills/training-check Command: npx skills add https://github.com/Mang30/myskills --skill training-check-mang30
It prevents wasted GPU compute by detecting early signs of broken or degrading training quality in Weights & Biases metrics (such as NaNs, loss divergence, and stalled or worsening evaluation results) instead of discovering the issue only after training ends.
Use the training-check skill for wandb run <entity>/<project>/<run_id> while training is running so it can automatically monitor and stop or continue based on metric health.
High-intent search queries and answers about installing and using this skill.
Automatically detect loss divergence by periodically sampling Weights & Biases metrics to monitor loss trends, NaN/Inf occurrences, and gradient norm stability, stopping bad runs early to save compute. The skill checks metric health at adaptive intervals during active training.
NaN or Inf values in training metrics indicate numerical instability or broken gradients, caught by periodic health checks that sample loss trends and gradient norms from Weights & Biases. The monitoring process identifies these anomalies and stops the run before more GPU hours are wasted.
You can monitor Weights & Biases runs without an active API connection because the skill provides fallback resilience by reverting to log-file inspection over SSH when the WandB API is unreachable. This ensures continuous monitoring of training quality even during network disruptions.
Stop a model training run automatically when evaluation metrics stall by configuring periodic health checks that monitor eval metric changes, learning-rate behavior, and loss trends. The skill safely decides to stop when metrics are clearly bad or escalate when ambiguous.
This training monitor supports adaptive checking intervals that scale from 10 to 60 minutes as run health improves, reducing unnecessary checks on stable runs. It periodically validates training quality by sampling loss trends, evaluation metrics, and gradient norm patterns.
lightrain-aMonitor WandB training metrics to detect loss divergence, NaN/Inf, and eval degradation.
THUFanZdMonitor WandB metrics to detect NaN, divergence, or idle GPUs.
gztcodeAutomate ML experiment tracking and collaboration with Weights & Biases.
ChuaHanChongPoll ML training log files to detect divergence and terminate failing processes.
