monitor-experiment

Monitor remote GPU experiments, collect results, and finalize cost manifests over SSH.

75|7|Updated May 2, 2026
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill monitor-experiment-zjunlp
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: monitor-experiment
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/monitor-experiment
Command: npx skills add https://github.com/zjunlp/Mechanist --skill monitor-experiment-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires wandb.

What problem does it solve? Tracking long-running machine learning experiments across SSH servers, Vast.ai instances, and Modal apps is tedious and error-prone. This Skill checks what is running, collects screen output and JSON results, pulls Weights & Biases metrics, and finalizes per-run cost records so you always know experiment status and spend. ## Core Features & Use Cases - Status Checks: Inspect screen sessions over SSH, query Vast.ai instance state, and list Modal apps to determine whether runs are active, finished, or crashed. - Result Collection: Capture screen output, fetch JSON result files, and pull training curves and eval metrics from the W&B API for comparison against baselines. - Cost Finalization: Write canonical cost.json manifests with wall-clock time, GPU hours, and run status for downstream budget tracking, plus cost reminders for Vast.ai and Modal. - Use Case: You launched three fine-tuning runs on a remote GPU server last night. Ask to check results, and get a comparison table of metrics versus baseline, W&B dashboard links, and a reminder to destroy idle Vast.ai instances. ## Quick Start Check the results of my running experiments on the remote server and summarize the metrics compared to baseline.

Frequently Asked Questions about monitor-experiment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I check if my remote training experiment is finished?

Run screen -ls over SSH to list active sessions, then capture the last lines of each screen with hardcopy to inspect progress. If the session is gone, check the results directory for JSON output files to confirm completion.

How to pull training metrics from Weights & Biases via API?

Use the wandb Python API to list runs in a project and call scan_history for metrics like train/loss and eval/loss. Run summary gives final metrics, and the run state shows whether it is running, finished, or crashed.

Can I monitor experiments running on Vast.ai or Modal?

Yes. For Vast.ai, read the SSH host and port from vast-instances.json and check instance status with vastai show instances. For Modal, use modal app list and modal app logs; finished apps disappear from the list automatically.

Why does my experiment show no JSON results after finishing?

Missing result files usually indicate a crashed run. Check the screen log for non-zero exits or tracebacks before concluding, and mark the run as failed when finalizing its cost.json manifest.

How is GPU cost tracked for completed experiment runs?

Each completed run gets a cost.json with wall-clock seconds, GPU hours computed from GPU count, and final status. For Vast.ai, report hours times the hourly rate and remind the user to destroy idle instances; Modal bills only for execution time.