monitor-experiment

Check running screen sessions over SSH with screen -ls, then capture the last lines of each session using screen hardcopy. Finished runs typically produce JSON result files you can fetch and parse for final metrics.

Use the wandb Python API to list runs in a project, scan run history for metrics like train/loss and eval/loss, and fetch the run summary for final values. This gives loss curves and convergence signals beyond raw screen output.

Yes. For Vast.ai, read the SSH host and port from vast-instances.json and check instance status with vastai show instances. For Modal, use modal app list and modal app logs; finished apps disappear from the list automatically.

Hardcopy can fail if the screen session ended or lacks permissions. In that case, fall back to checking log files or tee output written by the training script, or look for JSON result files in the results directory.

Track running cost by multiplying elapsed hours by the hourly rate stored in vast-instances.json. When all experiments on an instance are done, destroy the instance to stop billing; Modal auto-scales to zero so no cleanup is needed there.