distributed-llm-pretraining-torchtitan

Pretrain LLMs with 4D parallelism by using torchtitan to compose FSDP2, Tensor Parallel, Pipeline Parallel, and Context Parallel together. This scales distributed training across large GPU clusters natively within PyTorch.

Yes, torchtitan supports multi-node distributed training on SLURM. It provides throughput-focused training recipes that manage scaling from single-node 8-GPU setups to larger SLURM-managed clusters for extensive LLM pretraining.

Yes, you can enable Float8 acceleration for LLM pretraining on H100-class hardware. The torchtitan recipes include specific configurations to leverage Float8 for faster distributed training throughput.

Distributed checkpointing uses PyTorch Distributed Checkpoint to enable fault-tolerant training and recovery. It supports resharding across different parallelism modes and handles seed checkpoint requirements for Pipeline Parallel resumption.

Start single-node 8-GPU pretraining by configuring your training TOML config for the target model size and executing the run_train.sh launcher with torchrun. This applies the torchtitan composable 4D parallelism setup.

A seed checkpoint is required for Pipeline Parallel resumption to ensure correct distributed checkpointing and recovery. It allows PyTorch Distributed Checkpoint to properly restore state when resharding across parallelism modes.