ml-training-monitor

Diagnose machine learning training runs by interpreting loss curves and validation metrics.

Updated Apr 13, 2026
One-click install
npx skills add https://github.com/henrycashe26/my_skills --skill ml-training-monitor
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: ml-training-monitor
Source: https://github.com/henrycashe26/my_skills/tree/main/ml/ml-training-monitor
Command: npx skills add https://github.com/henrycashe26/my_skills --skill ml-training-monitor

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This skill addresses the uncertainty and technical debt associated with training machine learning models, helping you identify whether a run is healthy, failing, or requires intervention.

Core Features & Use Cases

  • Diagnostic Workflow: Provides a structured approach to observing loss curves, instrumenting metrics, and making evidence-based decisions on hyperparameter adjustments.
  • Symptom-Based Debugging: Offers a comprehensive lookup table for common training failures like NaN loss, divergence, OOM errors, and vanishing gradients.
  • Use Case: When your model loss spikes unexpectedly or validation metrics diverge, use this skill to interpret your logs and determine if you should adjust the learning rate, modify the architecture, or kill the run to save compute.

Quick Start

Use the ml-training-monitor skill to analyze the training logs I have provided and suggest a diagnosis and fix.

Frequently Asked Questions about ml-training-monitor

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug NaN loss and vanishing gradients during model training?

Debug NaN loss and vanishing gradients using symptom-based diagnostics to interpret training logs and identify evidence-based fixes like adjusting learning rates or modifying model architecture.

What do unstable loss curves and validation metric divergence indicate during deep learning training?

Unstable loss curves and validation metric divergence indicate training instability, requiring systematic evaluation of gradient norms and validation metrics to determine if hyperparameter adjustments or run termination are necessary.

How do I interpret training logs to decide whether to adjust hyperparameters or kill a run?

Interpret training logs by observing loss curves and instrumenting metrics to evaluate convergence and resource utilization, enabling evidence-based decisions on whether to adjust hyperparameters or kill the run to save compute.

When should I use systematic training diagnostics for machine learning model debugging?

Use systematic training diagnostics for machine learning model debugging when validation metrics diverge or loss spikes unexpectedly, ensuring structured evaluation of training stability before committing further compute resources.

What is the best way to monitor training stability and convergence across deep learning hyperparameter tuning scenarios?

Monitor training stability and convergence by applying a structured diagnostic workflow to observe loss curves and gradient norms, yielding systematic evaluation of model training runs during hyperparameter tuning.

Can I diagnose out of memory errors and training divergence without external dependencies?

Yes, you can diagnose out of memory errors and training divergence without external dependencies by using a comprehensive symptom lookup table to interpret logs and determine necessary architecture modifications.