V-JEPA 2 Distributed Training

What problem does it solve?

Distributes and scales V-JEPA 2 training across multiple GPUs and nodes, enabling efficient resource usage and fault tolerance in large-scale experiments.

Core Features & Use Cases

• NCCL-based distributed setup with auto-detection of SLURM or local multi-GPU; supports preemption-safe checkpointing and code snapshotting
• SLURM integration with Submitit for fault-tolerant job scheduling and automatic requeue
• Reference implementations for distributed utilities (checkpoint management, custom autograd distributed ops, and prefix-stripping for pretrained weights)

Quick Start

Launch a distributed V-JEPA 2 training job across multiple GPUs with NCCL and SLURM, including preemption-safe checkpointing.