distributed-llm-pretraining-torchtitan

Pretrain LLMs with 4D parallelism by using torchtitan to combine FSDP2, tensor parallel, pipeline parallel, and context parallel across large GPU counts. You configure parallelism and training settings via TOML job configs and launch runs with torchrun.

Yes, Float8 and torch.compile are supported for distributed LLM pretraining. Float8 is enabled on H100-class hardware to optimize performance, while torch.compile accelerates execution alongside distributed checkpointing.

SLURM is not strictly required but is supported for multi-node LLM pretraining with torchtitan. The Skill covers single-node runs, multi-node SLURM runs, checkpoint resumption, and large-scale 405B-style 4D parallel configurations.

4D parallel pretraining requires PyTorch 2.6.0 or higher, torchtitan 0.2.0 or higher, and torchao 0.5.0 or higher. These dependencies provide the native FSDP2 and composable parallelism features needed for scalable training.

Start pretraining Llama 3.1 on 8 GPUs by running torchrun with 8 processes and passing the Llama 3 8B TOML config file to the torchtitan train module. This initiates the distributed training job using your specified settings.

Yes, pipeline parallelism requires seed checkpoints for distributed training. Torchtitan explicitly supports seed checkpoint requirements for PP to ensure correct and repeatable distributed training workflows across large GPU counts.