launch-nemo-rl

To launch NeMo-RL training on Kubernetes, use the nrl-k8s CLI to submit an infra YAML recipe pair. You can execute a single command to start an experiment and begin iterative monitoring immediately.

Ephemeral mode supports one-off NeMo-RL experiments, while long-lived mode enables iterative development. Both modes manage the RayCluster lifecycle and allow you to monitor status, retrieve logs, and debug training jobs.

Yes, kubectl-based observability is supported for monitoring NeMo-RL experiments on Kubernetes. You need access to a Kubernetes cluster, an infra YAML pair for recipes, and the nrl-k8s CLI to manage and observe RayCluster resources.

Debug hung or failed NeMo-RL training jobs by retrieving logs and checking status through the nrl-k8s CLI. The playbook covers end-to-end debugging, including stopping, tearing down, and iterating on failed RayCluster resources.

The best way to manage the RayCluster lifecycle is using the nrl-k8s CLI on Kubernetes. It provides a repeatable approach to submit recipes, monitor training status, access logs, and tear down RayCluster resources after experiments.

An infra YAML pair is required for Kubernetes deployment to define the NeMo-RL recipe configuration. It provides the necessary specifications for the nrl-k8s CLI to orchestrate RayCluster resources, ensuring reproducibility and governance for teams.