monitor-cron-sweep

Cross-cluster job status monitoring consolidates squeue and sacct data from multiple clusters into a single unified report. It buckets jobs by type, such as RL, SFT, datagen, and eval, rendering a standardized table for tracking ongoing experiments.

Consolidating squeue and sacct data collects running and terminal state information across clusters, buckets jobs by type, and produces a deterministic job_monitor_table.md file. This standardized output supports tracking, auditing, and cross-cluster visibility.

Cluster job monitoring identifies silent hangs, inode headroom issues, and per-type red flags by gathering queue and terminal state data. It flags completions or failures per job type and proposes appropriate cleanup or investigation steps.

Tracking RL, SFT, datagen, and eval jobs across clusters requires bucketing running and terminal states by type. The monitoring process renders a consolidated cross-cluster table, applying health flags and cleanup handoffs for each specific job category.

Multi-cluster job monitoring requires no external dependencies or components to function. It directly gathers squeue and sacct data, verifies liveness, and renders cross-cluster tables to flag completions or failures per type.

Flagging completions and failures for multi-cluster experiments involves collecting queue data and terminal states, then bucketing jobs by type. This approach identifies per-type red flags and generates cleanup handoffs within a standardized report.