monitor-experiment

To check if model training is done on an SSH server, you can monitor active screen sessions to read recent outputs and locate JSON result files for a quick status summary. This process extracts training progress and flags unexpected outcomes like NaNs.

Yes, you can optionally pull Weights and Biases metrics when monitoring a running experiment to retrieve run state, training curves, and final summaries. The process queries W&B via Python and provides dashboard links alongside the collected JSON results.

Experiment monitoring works with Vast.ai GPU instances and Modal apps by checking instance status and execution time. It gathers latest outputs and JSON artifacts to report training progress, incorporating cost-awareness for Vast.ai and execution-time awareness for Modal.

The best way to collect JSON results and training logs from multiple screen sessions is to automate the extraction of recent screen output and inspect a designated results directory. This produces a summarized comparison table detailing training progress and outcomes.

To get a summary notification when your remote training experiment finishes, configure an optional Feishu notification integration. When enabled and a Feishu config exists, it automatically sends an experiment completion summary with the latest metrics.

Your experiment monitoring report shows NaNs or divergence because the automated summarization process specifically flags unexpected outcomes found in the collected JSON artifacts or W&B metrics. It highlights these training anomalies and suggests appropriate next steps.