launch-nemo-rl

Launch NeMo-RL recipes on Kubernetes using the nrl-k8s CLI to submit either an ephemeral RayJob or a long-lived RayCluster. You must provide correctly paired recipe and infra YAML files along with appropriate submitter and code-source options.

If your RayJob is stuck or failing, you can use nrl-k8s CLI debugging commands to validate state with check, inspect status, and list or fetch role logs. You can also query Ray dashboard APIs for deeper operational observability during post-mortem troubleshooting.

A RayJob is ephemeral and tears down automatically after one-shot execution, whereas a long-lived RayCluster allows iterative development loops and reusable deployments. You can reuse clusters when specs drift and manage deployments alongside them using the nrl-k8s CLI.

Yes, you can test NeMo-RL recipe changes without forking by applying Hydra entrypoint overrides. This config-safe iteration approach uses the correct infra and recipe YAML pairs to validate modifications dynamically through the nrl-k8s CLI.

To retrieve post-mortem logs for failed NeMo-RL training, use the nrl-k8s CLI job and role commands to list and fetch logs. You can also inspect the Ray dashboard APIs and validate previous state using the check command for comprehensive debugging.

Yes, running NeMo-RL requires correctly paired recipe and K8s/Ray infra YAML files. Careful selection of submitter, code-source, and wait options is necessary to validate state, monitor roles, and safely tear down resources using nrl-k8s.