distributed-llm-pretraining-torchtitan

Configure torchtitan TOML jobs for 4D parallel LLM pretraining on GPUs.

Updated Apr 9, 2026

Search Tags:#checkpointing #distributed training #torchrun #float8 #fsdp2 #llm pretraining #torch titan

Ready to run

npx skills add https://github.com/MarbleSodas/Mavis --skill distributed-llm-pretraining-torchtitan-marblesodas

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/MarbleSodas/Mavis/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/MarbleSodas/Mavis --skill distributed-llm-pretraining-torchtitan-marblesodas

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0, and includes references (resource) components.

What problem does it solve?

Provides a practical, PyTorch-native way to pretrain large language models across many GPUs without hand-wiring complex distributed training setups.

Core Features & Use Cases

4D parallelism with TorchTitan: Uses composable FSDP2, tensor parallel (TP), pipeline parallel (PP), and context parallel (CP) for scalable pretraining.
Performance-oriented training knobs: Supports Float8 on H100-class hardware, torch.compile, and distributed checkpointing.
Repeatable distributed workflows: Covers single-node, multi-node (SLURM), checkpoint resumption, and seed checkpoint requirements for PP.

Quick Start

Start pretraining Llama 3.1 on 8 GPUs by running: torchrun --nproc_per_node=8 -m torchtitan.train --job.config_file ./torchtitan/models/llama3/train_configs/llama3_8b.toml

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I pretrain an LLM with 4D parallelism using PyTorch?▼

Pretrain LLMs with 4D parallelism by using torchtitan to combine FSDP2, tensor parallel, pipeline parallel, and context parallel across large GPU counts. You configure parallelism and training settings via TOML job configs and launch runs with torchrun.

Can I use Float8 and torch.compile for distributed LLM pretraining?▼

Yes, Float8 and torch.compile are supported for distributed LLM pretraining. Float8 is enabled on H100-class hardware to optimize performance, while torch.compile accelerates execution alongside distributed checkpointing.

Do I need SLURM to run multi-node LLM pretraining with torchtitan?▼

SLURM is not strictly required but is supported for multi-node LLM pretraining with torchtitan. The Skill covers single-node runs, multi-node SLURM runs, checkpoint resumption, and large-scale 405B-style 4D parallel configurations.

What are the PyTorch and torchtitan version requirements for 4D parallel pretraining?▼

4D parallel pretraining requires PyTorch 2.6.0 or higher, torchtitan 0.2.0 or higher, and torchao 0.5.0 or higher. These dependencies provide the native FSDP2 and composable parallelism features needed for scalable training.

How do I start pretraining Llama 3.1 on 8 GPUs?▼

Start pretraining Llama 3.1 on 8 GPUs by running torchrun with 8 processes and passing the Llama 3 8B TOML config file to the torchtitan train module. This initiates the distributed training job using your specified settings.

Does pipeline parallelism require seed checkpoints for distributed training?▼

Yes, pipeline parallelism requires seed checkpoints for distributed training. Torchtitan explicitly supports seed checkpoint requirements for PP to ensure correct and repeatable distributed training workflows across large GPU counts.