babysit-training

Monitor ML training jobs and auto-restart from the last checkpoint on crash.

Updated Mar 13, 2026
One-click install
npx skills add https://github.com/Albatross679/0313temp --skill babysit-training
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: babysit-training
Source: https://github.com/Albatross679/0313temp/tree/main/.claude/skills/babysit-training
Command: npx skills add https://github.com/Albatross679/0313temp --skill babysit-training

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Monitor and safeguard ongoing ML training jobs by performing comprehensive health checks and auto-restart from the last checkpoint on crash. Document any issues found in issues/ with proper frontmatter.

Core Features & Use Cases

  • Continuous health checks: process alive, log progression, GPU utilization, disk space, checkpoint integrity, and W&B connectivity.
  • Auto-restart from last checkpoint on crash and automatic issue documentation in issues/ with frontmatter.
  • Periodic, loop-based monitoring for uninterrupted training visibility and rapid incident response.

Quick Start

Start monitoring by initiating the babysit-training loop to continuously observe training status and auto-recover when needed.

Frequently Asked Questions about babysit-training

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I monitor ML training runs and automatically restart from the last checkpoint on crash?

To monitor ML training runs and auto-restart on crash, initiate a continuous monitoring loop that performs health checks and resumes training from the last checkpoint. This ensures uninterrupted training sessions.

What health checks are needed for safeguarding long-running T5 model training jobs?

Safeguarding T5 model training jobs requires health checks for process status, log progression, GPU utilization, disk space, checkpoint integrity, and W&B connectivity. Periodic monitoring provides rapid incident response.

Does W&B integration support monitoring hyperparameter sweeps for NL-to-SQL projects?

Yes, W&B integration supports monitoring hyperparameter sweeps for NL-to-SQL projects. The monitoring loop verifies W&B connectivity and tracks metric trending across single runs and long-running sessions.

How do I document training issues found during GPU monitoring and log analysis?

To document training issues found during GPU monitoring and log analysis, automatically write files into an issues directory with proper frontmatter. This creates a structured status output for incident tracking.

What is the best way to track metric trending and disk health during hyperparameter sweeps?

The best way to track metric trending and disk health during hyperparameter sweeps is using a periodic, loop-based monitoring tool. It continuously observes training status and safeguards ongoing jobs.