rlxp-monitor-training

You monitor RL training runs by inspecting process status, GPU assignment, queue checks, and metric streams to detect NaN/Inf events, simulator crashes, and saturation. This produces a continue, wait, or stop-investigate recommendation based on run health.

Simulator crashes and reward domination during reinforcement learning stem from instability, resource issues, and metric regressions. Run monitoring checks process state, GPU assignment, and logging behavior to flag these early warning signs before they corrupt a study.

You check for failing RL training runs by monitoring logs, GPU utilization, metric streams, and guardrails for regressions. Health monitoring detects episode collapse, saturation, and missing metrics to catch instability early and prevent wasted compute.

You cannot automatically stop RL training when guardrails are violated. Run monitoring produces a stop-investigate recommendation but does not exceed its contract authority, meaning it advises action without unauthorized run termination.

When GPU utilization drops unexpectedly during RL training, you should inspect process status, GPU assignment, and queue status. Monitoring flags saturation, missing metrics, and suspicious logging behavior to determine whether to continue, wait, or stop and investigate.

RL run monitoring works with active training jobs that have logs, process status, GPU assignment, and metric streams. It applies to in-progress task and study runs across simulator, logging, and reward-heavy workflows to detect failures before they spread.