distributed-llm-pretraining-torchtitan

What problem does it solve?

This Skill addresses the challenge of efficiently pretraining large language models (LLMs) at scale, enabling users to train models from 8 billion to over 512 billion parameters using advanced parallelism techniques.

Core Features & Use Cases

• 4D Parallelism: Leverages FSDP2, Tensor Parallelism (TP), Pipeline Parallelism (PP), and Context Parallelism (CP) for optimal resource utilization.
• Optimized Training: Supports Float8 precision for significant speedups on H100 GPUs, along with torch.compile for fused kernels.
• Model Support: Designed for pretraining models like Llama 3.1, DeepSeek V3, and custom architectures.
• Use Case: A research team needs to pretrain a new 70B parameter LLM on a cluster of 256 GPUs. They can use this Skill to configure and launch the training job, managing the complex parallelism and optimization strategies.

Quick Start

Use the distributed-llm-pretraining-torchtitan skill to launch Llama 3.1 8B pretraining on 8 GPUs using the default configuration file.