babysit-zephyr

Monitor a Zephyr job by checking child job states with the Iris CLI list-tasks command, then reading coordinator logs for stage, completed, in-flight, and queued counts. Start with a smoke check in the first 2-5 minutes, then calibrate check-in intervals to observe stage transitions.

Start a job with uv run iris --config <CONFIG> job run --region <REGION> --no-wait -- python <SCRIPT>. For long-running pipelines that accumulate state, add --memory 5GB since the entrypoint container defaults to only 1GB.

The Terminated by user message is misleading and does not necessarily mean a human killed the job. The system uses it for various internal termination reasons, so always check the actual logs at the parent job, coordinator, and worker levels to find the real cause.

Request a thread dump of the coordinator using the profile-task RPC with a threads profile type. If all threads are in _worker with the thread pool idle, the pipeline has exited and the coordinator is a zombie; a live pipeline shows actor-method_0 in _wait_for_stage.

Restart only after asking the user for explicit approval, since stopping kills all child jobs including coordinators and workers. First confirm the failure is real by checking the latest attempt's logs, then stop the job, reuse or obtain the run command, and resume monitoring.