manage-hero-run

Start from the reference launcher, confirm the source commit ancestry, pin a unique output root under a user-owned path, validate checkpoint retention and W&B tracker settings, and print the exact submit command before launching. Assign a DRI and arrange babysitting unless explicitly declined.

Relaunch directly with the same run id, W&B identity, and output root for recoverable failures like preemption or hardware issues. Select the newest complete checkpoint with metadata.json, sorted by parsed numeric step, and never use incomplete checkpoints.

Use a new run id and W&B identity when the old lineage is unsafe or semantically different, such as W&B corruption or nontrivial code changes to the model, training loop, optimizer, or data pipeline. Document old and new identities in the issue and logbook.

A production run is long-lived, has a durable output contract, or exceeds roughly 1e22 FLOPs, requiring a full run record with issue and logbook. A bounded diagnostic has fixed step limits and lifecycle-managed output, recorded only in the originating conversation.

Levanter resumes are generally bitwise identical on TPU, but GPU runs can differ slightly. Alert if loss differs by more than 0.002 during catch-up or more than 1% after post-resume warmup, as larger gaps indicate lineage or config problems.

Keep one rolling temporary resume checkpoint by default and never more than five, with rollback depth above two requiring recorded rationale. Store resume checkpoints in a lifecycle-managed temp prefix and canonical exports under a durable user-owned path.