V-JEPA 2 Video & Masking

Spatiotemporal masking for video pretraining is implemented using a multi-block 3D algorithm that partitions the token grid into encoder and prediction masks. This guarantees no overlaps or gaps, enabling structured target creation for V-JEPA 2 training.

Tubelet tokenization via PatchEmbed3D converts video clips into efficient spatiotemporal embeddings. It processes temporal and spatial dimensions together to generate structured token grids required for V-JEPA 2 pretraining.

Variable-length video clips are batched using a MaskCollator that groups sequences by frames per clip (FPC). This enables memory-conscious training by wrapping heterogeneous clip lengths with MultiSequenceEncoder and MultiSequencePredictor.

PyTorch supports deterministic seed handling across workers for video masking by enforcing invariants during mask generation. This ensures encoder and prediction masks partition the token grid consistently across distributed training processes.

Encoder and prediction masks overlap when partition invariants are not enforced during multi-block 3D mask generation. Proper spatiotemporal masking algorithms ensure the token grid is partitioned without overlaps or gaps to prevent this.

Multi-block 3D masking supports variable-length video clips through a MaskCollator that groups batches by frames per clip. MultiSequenceEncoder and MultiSequencePredictor wrappers handle heterogeneous clip lengths in a single training step.