supervising-training-runs

What problem does it solve?

It helps you launch and monitor long training jobs without wasting hours on runs that have already gone wrong, catching NaNs, stalls, label bugs, throughput collapse, and dead processes early.

Core Features & Use Cases

• Preflight validation: Bench the real workload, audit the data, and run a small smoke test before committing to a full GPU window.
• Structured telemetry: Log per-step and per-eval metrics to JSONL so progress can be checked with quick tail reads instead of noisy log dumps.
• Watchdog supervision: Run a separate monitor that exits on anomalies like non-finite loss, stale metrics, or chance-level evaluation.
• Failure recovery: Use checkpoints, completion sentinels, and fail-loud behavior to make long jobs safer and easier to resume.
• Use case: A fine-tune is launched overnight, the telemetry file shows val AUROC staying near chance, and the watchdog alerts you before the GPU window is lost.

Quick Start

Ask Claude to set up a long training run with structured JSONL metrics, a watchdog process, and early anomaly alerts.