distributed-llm-pretraining-torchtitan

Download the tokenizer with the download_hf_assets script, create a TOML config specifying the llama3 model and 8B flavor, then launch with torchrun using 8 processes per node. Training logs to TensorBoard and checkpoints save at configured intervals.

Torchtitan is PyTorch-native with composable 4D parallelism and no third-party dependencies, ideal for pretraining from scratch. Megatron-LM targets maximum NVIDIA-only performance, while DeepSpeed offers a broader ZeRO optimization ecosystem and inference support.

Yes, torchtitan supports Float8 training on H100 or newer GPUs via torchao with tensorwise or rowwise scaling. Combined with torch.compile, benchmarks show up to 48% throughput improvement over FSDP baselines on Llama 8B.

Enable full activation checkpointing instead of selective mode and reduce local batch size in the TOML config. You can also use gradient accumulation by setting a larger global batch size while keeping local batch size at 1.

Yes, torchtitan supports saving directly in HuggingFace format during training or offline conversion using the convert_to_hf and convert_from_hf scripts. Checkpoints saved with last_save_model_only also load directly into torchtune for fine-tuning.

Pipeline parallelism requires a seed checkpoint created on a single device to ensure consistent initialization across all PP stages. Run one training step with all parallelism degrees set to 1 and create_seed_checkpoint enabled before launching the full job.