Monitor WandB metrics to detect NaN, divergence, or idle GPUs.
npx skills add https://github.com/THUFanZd/Sewed_pipeline --skill training-check-thufanzd
Please help me install this Agent Skill. Skill: training-check Source: https://github.com/THUFanZd/Sewed_pipeline/tree/main/.agents/skills/training-check Command: npx skills add https://github.com/THUFanZd/Sewed_pipeline --skill training-check-thufanzd
Periodically read WandB metrics during training to catch problems early. Do not wait until training finishes to discover it was a waste of GPU time.
Configure a recurring job to run training-check for your WandB run every 10 minutes.
High-intent search queries and answers about installing and using this skill.
Monitor WandB metrics during ML training by periodically reading run history to detect NaN, loss divergence, or idle GPUs, catching problems early instead of waiting for training to finish.
Automate GPU training health checks by configuring a recurring job to read metrics every 10 minutes, which adjusts its interval over time based on stability from 10 to 60 minutes.
When the WandB API network is unavailable, the training health check falls back to SSH log access, using tail log commands to read training status and ensure monitoring continuity.
This automated training monitoring focuses specifically on training quality signals like NaN and loss divergence, rather than process health, and is designed for automated recurring checks.
Automated ML training health checks handle ambiguous signals using a built-in decision workflow with escalation rules, ensuring uncertain metrics are properly evaluated before stopping a run.
JuanMS20Configure GitHub authentication for Hermes Agent using git, gh CLI, or SSH credentials.
xqinagMonitor WandB metrics during training to detect NaN, divergence, or idle GPUs.
nishide-devExecute PyTorch Lightning and Hydra training runs with monitoring, checkpointing, and W&B integration.
KwongFukDetect WandB metric anomalies during active model training sessions.
