V-JEPA 2 Distributed Training

Scale V-JEPA 2 training across multiple GPUs and nodes using NCCL-based distributed setup with SLURM integration via Submitit for fault-tolerant job scheduling and automatic requeue. It auto-detects SLURM or local multi-GPU environments to initialize distributed training.

Distributed checkpointing handles preemption by implementing preemption-safe checkpoint management with retry logic. This ensures training progress is saved robustly, allowing V-JEPA 2 jobs to recover and resume automatically after interruptions or SLURM requeues.

You need custom autograd distributed operations for gradient-flow-correct AllGather operations in V-JEPA 2. The Skill provides reference implementations for these custom ops to ensure gradients flow correctly across GPUs during distributed training.

You can automatically strip prefixes when loading pretrained weights in PyTorch. The Skill includes reference utilities for pretrained weight loading with automatic prefix stripping, ensuring compatibility between checkpoint keys and model state dictionaries.

Ensure reproducibility in distributed V-JEPA experiments by utilizing the built-in code snapshotting feature. This captures the exact code state alongside preemption-safe checkpointing, guaranteeing that large-scale training runs can be reliably reproduced or resumed.