What problem does it solve? Pretraining large language models from scratch requires coordinating complex distributed training across hundreds of GPUs, and misconfiguring parallelism strategies, checkpointing, or Float8 quantization wastes expensive compute time. ## Core Features & Use Cases - 4D Parallelism Configuration: Compose FSDP2, Tensor Parallel, Pipeline Parallel, and Context Parallel strategies for models from 8B to 405B+ parameters. - Float8 Training on H100s: Enable rowwise or tensorwise Float8 scaling via torchao for 30-50% throughput speedups with layer filtering. - Checkpoint Management: Use PyTorch Distributed Checkpoint for fault-tolerant saves, seed checkpoints for pipeline parallelism, and HuggingFace/torchtune conversion. - Use Case: Pretrain Llama 3.1 70B across 32 SLURM-managed nodes by selecting a registered config, setting FSDP and TP degrees, submitting the job, and resuming automatically from checkpoints. ## Quick Start Use the torchtitan skill to pretrain a Llama 3.1 8B model on 8 GPUs with the llama3_8b config and TensorBoard monitoring.
