skills.rest
duypham2801
duypham2801COMMUNITY
Explore all skills by this author →
GitHub

training-check

Analyze WandB metrics to detect NaNs, loss divergence, and evaluation degradation.

Updated May 25, 2026
Search Tags:#deep learning#wandb#training monitoring#cron automation#loss divergence#nan detection#gpu runs
One-click install
npx skills add https://github.com/duypham2801/ThS_LLM --skill training-check-duypham2801
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: training-check
Source: https://github.com/duypham2801/ThS_LLM/tree/main/.claude/skills/training-check
Command: npx skills add https://github.com/duypham2801/ThS_LLM --skill training-check-duypham2801

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you detect training-quality problems early (like NaNs, divergence, or worsening eval metrics) so you can stop or adjust a run before you waste more GPU hours.

Core Features & Use Cases

  • WandB metric health monitoring: Periodically reads run history to assess loss trends, learning-rate behavior, gradient health, and eval movement.
  • Robust decisioning with escalation: Automatically distinguishes clearly bad vs clearly fine outcomes, and only escalates to Codex when the signal is ambiguous.
  • Fallback when WandB is unreachable: Uses SSH-tail of a training log as an alternative source to continue the assessment.

Use Case: You start a long semi-supervised training run, and after a few intervals you notice loss divergence or metric degradation in WandB—this Skill helps you decide whether to stop or keep going.

Quick Start

Use the training-check Skill to periodically review the WandB run history for your run_id and decide whether to stop, wait, or continue based on the observed trends.

Frequently Asked Questions about training-check

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor WandB training runs for loss divergence and NaNs?▼

You can monitor WandB training runs for loss divergence and NaNs by analyzing run history metrics to assess loss trends, learning-rate behavior, and gradient health. This Skill periodically reads run history to detect training-quality issues early.

Can I automate WandB metric checks during long-running GPU training?▼

Yes, you can automate WandB metric checks during long-running GPU training using scheduled cron automation. The Skill periodically reviews run history at set intervals to evaluate whether to stop, wait, or continue the deep learning training job.

What happens if WandB is unreachable during deep learning training monitoring?▼

If WandB is unreachable during deep learning training monitoring, the Skill falls back to inspecting training logs via SSH. It tails the training log as an alternative source to continue the assessment and evaluate training quality.

How does this Skill decide whether to stop or continue a training run?▼

The Skill decides whether to stop or continue a training run by distinguishing clearly bad versus fine outcomes from WandB metric trends. It escalates to Codex judgment only for ambiguous cases while prescribing stop, continue, or wait actions.

What do I need to identify my WandB run for training quality checks?▼

You need to provide the entity, project, and run_id to identify your WandB run for training quality checks. The Skill uses this identification to access run history and perform trend checks on loss, gradients, and evaluation metrics.

Related Skills

Explore More Education & Research Skills→
lightrain-alightrain-a

training-check

Monitor WandB training metrics to detect loss divergence, NaN/Inf, and eval degradation.

Community
Advanced
HeXiao-55HeXiao-55
1

training-check

Monitor WandB metrics to detect NaN, divergence, or stalled GPU utilization.

Community
Intermediate
rocklambrosrocklambros
2

scaffolding-pytorch-training-loop

Scaffold a PyTorch training loop with deterministic seeding, AMP, and checkpoint resume.

Community
Advanced
TabithaFannyTabithaFanny

training-check

Monitor WandB training metrics for NaN values, loss divergence, and idle GPUs.

Community
Advanced