Poll ML training log files to detect divergence and terminate failing processes.
npx skills add https://github.com/ChuaHanChong/ml-optimizer --skill monitor-chuahanchong
Please help me install this Agent Skill. Skill: monitor Source: https://github.com/ChuaHanChong/ml-optimizer/tree/main/skills/monitor Command: npx skills add https://github.com/ChuaHanChong/ml-optimizer --skill monitor-chuahanchong
This Skill prevents wasted computational resources and time by automatically detecting and terminating machine learning training experiments that show signs of divergence, such as NaN values, exploding losses, or plateaus.
Monitor the running experiment with ID 'exp-001' for loss divergence.
High-intent search queries and answers about installing and using this skill.
Automatically stop diverging machine learning training experiments by using a monitoring script that polls log files for NaN values, exploding losses, or plateaus and terminates the runaway process to prevent wasted compute resources.
Detect NaN values and exploding losses in PyTorch training logs by continuously polling the log files and applying configurable thresholds to identify common failure modes like loss explosion or metric plateaus during the run.
You can use a monitoring script to track multiple hyperparameter tuning runs, acting as a safety net that automatically kills any experiment producing nonsensical results so you can focus compute on promising runs.
You need PyTorch and PyYAML installed to use this monitoring capability, as the Python scripts rely on these dependencies for parsing logs and applying configurable thresholds for divergence detection.
Prevent wasted computational resources during ML training by deploying an automated monitor that identifies divergence signals like metric plateaus or loss explosions in real-time and kills the failing processes immediately.
Detecting training divergence using log file polling requires the running experiment to output metrics to log files and depends on configurable thresholds, meaning delayed logging or poorly set thresholds might affect detection speed.
fikriafTrack ML experiments, visualize training, and manage model registries with Weights & Biases.
fcamacholMonitor registration process statuses and trigger change notifications.
DifanaDAPAutomate ML experiment tracking and hyperparameter sweeps via the Python API.
thisismynewfmail-uiLog machine learning experiment metrics and visualize training progress with W&B.
