rl-job-cleanup

Automated checkpoint publishing uploads your RL training artifacts to Hugging Face after a job terminates. The workflow selects the best checkpoint using a trailing-5 EMA of reward, flattens weights to the repo root, and uploads them under laion/<job_name>-<step>-<size>.

Best RL experiment steps are selected by calculating a trailing-5 EMA of reward across all restart links. This identifies the highest-performing checkpoint to flatten to the repository root and publish, ensuring optimal model preservation.

RL artifact publication includes a secret scan before uploading weights to Hugging Face. It also registers the run in a Supabase database with cross-user safety checks and uploads training traces to support auditing.

You can clean up the experiments directory and cancel pending retries after an HPC RL run. The workflow cancels pending retries to avoid wasted compute, then removes the experiments directory after successful publication to reclaim storage.

Preserving Hugging Face RL artifacts involves uploading the best checkpoint weights alongside parsed training traces. The workflow re-uploads debugging logs and metrics to support post-run analysis and maintain full reproducibility across the restart chain.