experiment-monitor

Monitor active experiments by reading logs and verifying process health.

7|1|Updated Mar 19, 2026

Search Tags:#automation #monitoring #logs #metrics #experiments #ml-ops

Ready to run

npx skills add https://github.com/Shiien/Self-Evolved-Research-Framework --skill experiment-monitor

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: experiment-monitor
Source: https://github.com/Shiien/Self-Evolved-Research-Framework/tree/main/skills/experiment-monitor
Command: npx skills add https://github.com/Shiien/Self-Evolved-Research-Framework --skill experiment-monitor

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates monitoring of active experiments by reading logs in logs/experiments/*.yaml, verifying process health, and surfacing up-to-date status to reduce manual chasing of stalled runs.

Core Features & Use Cases

Load active experiments from logs/experiments/*.yaml and monitor their status.
Check process liveness locally or remotely, tail recent logs, and extract key metrics (loss, accuracy, epoch, step).
Detect errors such as OOM, NaN, or tracebacks; update status to running/completed/failed; optionally notify.

Quick Start

Monitor all active experiments and report their latest status in a single glance.

Frequently Asked Questions about experiment-monitor

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I monitor active ML experiments and auto-update their status?▼

You can monitor active ML experiments by reading logs from logs/experiments/*.yaml, verifying process health locally or via SSH, and automatically updating status to running, completed, or failed in logs/experiments/{exp_id}.yaml.

Can I detect OOM, NaN, or tracebacks in concurrent experiment runs?▼

Yes, detecting OOM, NaN, and tracebacks is supported for multiple concurrent runs. The Skill tails recent logs, extracts key metrics, and marks the experiment status as failed when such errors are found.

How do I check process liveness for remote experiments over SSH?▼

Checking process liveness for remote experiments is done by querying process status via SSH. The Skill connects to remote setups, verifies if the experiment process is still running, and updates the corresponding YAML status file accordingly.

Do I need a specific directory structure to monitor experiment logs?▼

Yes, you need a specific directory structure. The Skill requires access to logs/experiments/*.yaml files to load active experiments and writes status updates back to logs/experiments/{exp_id}.yaml, so this path must exist.

What is the best way to extract loss and accuracy metrics from experiment logs?▼

The best way to extract loss and accuracy metrics is by tailing recent experiment logs. The Skill automatically parses the logs to surface key metrics like loss, accuracy, epoch, and step for each active run.

Why are my experiment runs stalling without updating status?▼

Experiment runs may stall without updating status due to silent failures. The Skill addresses this by actively checking process health and detecting errors like OOM or NaN to ensure stalled runs are marked as failed.