monitor-experiment

Monitor W&B training runs to detect anomalies and generate alerts.

3.9k|398|Updated Oct 24, 2024
One-click install
npx skills add https://github.com/hao-ai-lab/FastVideo --skill monitor-experiment-hao-ai-lab
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: monitor-experiment
Source: https://github.com/hao-ai-lab/FastVideo/tree/main/.agents/skills/monitor-experiment
Command: npx skills add https://github.com/hao-ai-lab/FastVideo --skill monitor-experiment-hao-ai-lab

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill enables continuous monitoring of W&B training experiments, providing structured alerts and progress reports to ensure model training stays on track.

Core Features & Use Cases

  • Real-time Metrics Tracking: Connects to W&B or reads local summaries to monitor training loss, gradient norms, step times, and validation outputs.
  • Anomaly Detection & Alerts: Evaluates conditions such as loss spikes, gradient issues, and progress stalls, issuing severity-graded alerts.
  • 30-Minute Quality Check: After half an hour, summarizes training progress, loss reduction, and validation outputs to guide decision-making in long training runs.
  • Use Case: An engineer running a large model training job can deploy this Skill to automatically detect issues early and decide whether to continue or halt.

Quick Start

Connect the Skill to an ongoing W&B run by specifying its run ID, set the polling interval, and choose alert conditions to receive timely updates.

Frequently Asked Questions about monitor-experiment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor W&B training runs for loss spikes and gradient issues?

To monitor W&B training runs, you connect to an ongoing run by specifying its run ID, set the polling interval, and choose alert conditions to receive severity-graded alerts for loss spikes and gradient issues.

What is the best way to track progress on long or distributed model training experiments?

The best way to track long training experiments is using automated monitoring that evaluates metrics, step times, and validation outputs, generating structured progress reports to guide your decision to continue or halt.

Can I use this to automatically check model training health after 30 minutes?

Yes, you can automatically check model training health after 30 minutes; the Skill summarizes training progress, loss reduction, and validation outputs to guide decision-making in long training runs.

Does W&B monitoring work with local summaries if the API is unavailable?

Yes, W&B monitoring works with local summaries; the Skill connects to W&B or reads local summaries to monitor training loss, gradient norms, step times, and validation outputs.

What types of anomalies can I detect during model training?

You can detect anomalies during model training by evaluating conditions such as loss spikes, gradient issues, and progress stalls, issuing severity-graded alerts to optimize your training workflows.