experiment-monitor

You can monitor active ML experiments by reading logs from logs/experiments/*.yaml, verifying process health locally or via SSH, and automatically updating status to running, completed, or failed in logs/experiments/{exp_id}.yaml.

Yes, detecting OOM, NaN, and tracebacks is supported for multiple concurrent runs. The Skill tails recent logs, extracts key metrics, and marks the experiment status as failed when such errors are found.

Checking process liveness for remote experiments is done by querying process status via SSH. The Skill connects to remote setups, verifies if the experiment process is still running, and updates the corresponding YAML status file accordingly.

Yes, you need a specific directory structure. The Skill requires access to logs/experiments/*.yaml files to load active experiments and writes status updates back to logs/experiments/{exp_id}.yaml, so this path must exist.

The best way to extract loss and accuracy metrics is by tailing recent experiment logs. The Skill automatically parses the logs to surface key metrics like loss, accuracy, epoch, and step for each active run.

Experiment runs may stall without updating status due to silent failures. The Skill addresses this by actively checking process health and detecting errors like OOM or NaN to ensure stalled runs are marked as failed.