ray-on-k8s-engineer

Install the KubeRay operator via Helm in its own namespace, then define RayCluster, RayJob, or RayService manifests with head and worker pod specifications. Use RayJob for ephemeral training runs and RayCluster for long-lived interactive work.

Define a trainable that reports metrics per iteration, a search space with log-uniform ranges, a search algorithm, and the ASHA scheduler for early stopping. Set resources_per_trial to share GPU workers and attach the MLflow callback so trials log under one parent run.

Ephemeral RayJobs give full library isolation and release GPUs at job end, making them the default for training pipelines. Permanent clusters suit interactive work and cached datasets but risk config drift, GPU leaks, and multitenancy issues, so recycle them weekly.

Yes, set num_gpus to a fraction like 0.25 in @ray.remote to pack small tasks onto one card. Ray enforces bookkeeping only, not memory isolation, so co-located tasks must fit within VRAM, and max_calls=1 can force worker restarts to release GPU memory.

The Ray dashboard on port 8265 shares its port with the Job API, which allows arbitrary code execution by design. Never expose it publicly; access it only via kubectl port-forward or an authenticated ingress, and restrict access with NetworkPolicies.

Fix the Ray metrics export port and create two PodMonitors, one for the head and one for workers, since no label spans a whole Ray cluster. Add Grafana alerts for pending tasks, object-store spill, and GPU utilization, plus application-level metrics via ray.util.metrics.