Verified:GitHub

distributed-llm-pretraining-torchtitan

Orchestrate PyTorch-native 4D parallelism for distributed LLM pretraining with torchtitan.

19|4|Updated Apr 22, 2026

Search Tags:#distributed training #pipeline parallelism #float8 #llm pretraining #torch titan #pytorch fsdp2 #distributed checkpointing

Ready to run

npx skills add https://github.com/carterwayneskhizeine/hermes-agent-windows-R --skill distributed-llm-pretraining-torchtitan-carterwayneskhizeine

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/carterwayneskhizeine/hermes-agent-windows-R/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/carterwayneskhizeine/hermes-agent-windows-R --skill distributed-llm-pretraining-torchtitan-carterwayneskhizeine

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It solves the challenge of efficiently pretraining large language models across many GPUs while keeping memory and throughput under control, especially when you need reproducible distributed training with PyTorch-native tooling.

Core Features & Use Cases

4D parallelism (FSDP2 + TP + PP + CP): Use composable distributed strategies for models that range from 8B to 405B+ parameters.
Throughput optimizations (Float8 + torch.compile): Accelerate H100-class training by combining Float8 quantization and compilation where it helps.
Distributed checkpointing with DCP: Enable fault-tolerant, interoperable checkpoints (including async checkpointing and HuggingFace conversion workflows).

Use case examples:

Pretrain a Llama-class model (e.g., Llama 3.1 8B or 70B) on a single node or multi-node cluster using TorchTitan’s torchtitan.train entrypoints.
Run long-context training that benefits from context parallelism (CP) and activation checkpointing.
Resume or convert distributed checkpoints for downstream training or model export.

Quick Start

Start distributed pretraining on multiple GPUs by running: "CONFIG_FILE=./torchtitan/models/llama3/train_configs/llama3_8b.toml ./run_train.sh".

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I pretrain LLMs using PyTorch 4D parallelism?▼

Pretrain LLMs with 4D parallelism by orchestrating FSDP2, tensor parallelism, pipeline parallelism, and context parallelism via torchtitan. This composable strategy scales training efficiently from 8-GPU single-node runs to 256+ multi-node GPU clusters.

What is the best way to accelerate H100 training with Float8 quantization?▼

Accelerate H100 training by combining Float8 quantization with torchao integration and torch.compile. These throughput optimizations maintain memory and performance control during large-scale pretraining.

Can I resume distributed training using PyTorch Distributed Checkpoint?▼

Resume distributed training using fault-tolerant PyTorch Distributed Checkpoint (DCP) workflows. DCP enables interoperable checkpoints, including asynchronous checkpointing and HuggingFace conversion for downstream export.

How do I run long-context training with context parallelism?▼

Run long-context training by enabling context parallelism alongside activation checkpointing. This approach manages memory constraints effectively when processing extended sequence lengths across distributed GPUs.

Does torchtitan work for pretraining Llama 3.1 models on a single node?▼

Yes, torchtitan works for pretraining Llama-class models like Llama 3.1 8B or 70B on a single node. Start distributed pretraining on multiple GPUs by executing torchtitan.train entrypoints via run_train.sh.