distributed-llm-pretraining-torchtitan

Pretraining LLMs from scratch is enabled by PyTorch-native distributed training with composable 4D parallelism including FSDP2, Tensor Parallelism, Pipeline Parallelism, and Context Parallelism. It scales efficiently for models ranging from 8B to over 405B parameters.

4D parallelism combines Tensor Parallelism, Pipeline Parallelism, Context Parallelism, and FSDP2 to scale LLM pretraining across massive GPU clusters. This composable PyTorch-native approach enables high throughput and efficient resource utilization for models up to 405B+ parameters.

Yes, Float8 precision is supported to achieve significant speedups on NVIDIA H100+ GPUs during distributed LLM pretraining. The framework also integrates torch.compile for fused kernels to maximize training throughput.

Yes, the distributed training framework provides interoperable checkpoints with HuggingFace and torchtune. This allows seamless integration and checkpoint conversion for models pretraining from scratch using 4D parallelism.

You can start Llama 3.1 8B pretraining on 8 GPUs using the provided default configuration file. This leverages composable 4D parallelism and FSDP2 to efficiently launch distributed training from scratch.

Pipeline Parallelism and Context Parallelism are needed when scaling LLM pretraining to massive models like 405B+ parameters. They work alongside Tensor Parallelism and FSDP2 to partition model layers and long sequences across GPUs for maximum throughput.