notebook-to-production

Follow a mechanical sequence: verify restart-and-run-all succeeds, parameterize varying values, extract cells into src/ modules, pin the environment to an image digest, and externalize data and credentials. Then convert to a K8s Job or CronJob for single steps, or an Argo CronWorkflow for multi-step pipelines.

Productionize when any graduation criterion holds: it runs on a cadence, others consume its output, silent failure has a cost, it is the sole source of truth for a model or dataset, or it needs cluster-scale compute. Notebooks failing all five should stay interactive but remain in git.

Use a Kubernetes CronJob for single-step parameterized jobs with the pinned container image. Use an Argo CronWorkflow when the pipeline has multiple steps like validate, prep, train, and evaluate, where each extracted module becomes a step with typed artifact inputs and outputs.

Common causes are hidden state from out-of-order cell execution, client and worker library version mismatches, local file paths instead of object storage URIs, and unpinned environments. Restart-and-run-all plus version checks like Dask's get_versions(check=True) catch these before scheduling.

Use Dask's KubeCluster via the Dask operator with worker groups and cluster.adapt for autoscaling, or connect to a long-lived RayCluster for Ray workloads. Access dashboards through kubectl port-forward on ports 8787 (Dask) or 8265 (Ray), never a public bind.

Dask workers spill to disk around 60% memory, stop accepting data near 80%, and are terminated near 95%. Fix this by resizing workers or rechunking so steady-state memory stays under 60%, persisting reused intermediates, and ensuring spill targets local NVMe rather than network storage.