distributed-llm-pretraining-torchtitan

Scale PyTorch LLM pretraining by applying composable 4D parallelism (FSDP2, tensor, pipeline, and context parallelism) to distribute large models across nodes. Configure your TOML file and launch the run_train.sh entrypoint to manage workloads from 8 to 512+ GPUs.

Composable 4D parallelism combines FSDP2, tensor parallelism, pipeline parallelism, and context parallelism to efficiently scale large language models across multi-node clusters. This approach maximizes throughput while managing memory constraints for models like Llama 3.1 405B.

Yes, you can accelerate H100 throughput by integrating Float8 and torch.compile during Llama 3.1 pretraining. Additional performance optimizations include activation checkpointing and microbench-informed filtering to maximize training efficiency.

Convert PyTorch Distributed Checkpoints (DCP) to HuggingFace formats using established interoperability patterns. This enables seamless recovery and downstream fine-tuning workflows for models pretrained with FSDP2 sharding.

Yes, torchtitan supports multi-node SLURM jobs for pretraining 70B+ models. It provides operational guidance and best-practice configurations to reliably execute large-scale distributed training across single-node and multi-node SLURM clusters.

Configure pipeline parallelism for 405B+ models by utilizing seed checkpoints and composable 4D parallelism. This approach ensures efficient scaling and fault tolerance while maintaining high throughput during large-scale distributed training.