skills.rest
JamesFincher
JamesFincherCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Configure TorchTitan for distributed LLM pretraining with 4D parallelism.

Updated May 4, 2026
Search Tags:#distributed training#torchtitan#float8#fsdp2#llm pretraining#tensor parallel#pipeline parallel
One-click install
npx skills add https://github.com/JamesFincher/gengar --skill distributed-llm-pretraining-torchtitan-jamesfincher
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/JamesFincher/gengar/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/JamesFincher/gengar --skill distributed-llm-pretraining-torchtitan-jamesfincher

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0, and includes references (resource) components.

What problem does it solve?

It helps you pretrain large language models at scale by orchestrating PyTorch-native distributed training with composable 4D parallelism, reducing friction compared to assembling FSDP/TP/PP/CP yourself across many GPUs.

Core Features & Use Cases

  • 4D parallelism for pretraining: Runs FSDP2 with tensor parallelism (TP), pipeline parallelism (PP), and context parallelism (CP) using TorchTitan’s PyTorch-native components.
  • Production-grade workflows: Supports single-node runs, multi-node SLURM submissions, resumable distributed checkpointing, and seed checkpoint creation for pipeline parallelism.
  • Performance and precision options: Enables Float8 training on H100-class hardware and integrates torch.compile for additional speedups.

Quick Start

Ask your AI to: “Use the torchtitan skill to pretrain Llama 3.1 8B on 8 GPUs, using the provided 8-gpu quick start steps and a custom TOML config with checkpointing enabled.”

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain an LLM with 4D parallelism using TorchTitan?▼

To pretrain an LLM with 4D parallelism using TorchTitan, you configure a TOML file to orchestrate composable FSDP2, tensor parallelism, pipeline parallelism, and context parallelism across single-node or multi-node GPU clusters.

What is composable 4D parallelism in distributed training?▼

Composable 4D parallelism combines FSDP2, tensor parallelism, pipeline parallelism, and context parallelism to distribute large language model pretraining across many GPUs, reducing friction compared to assembling each component manually.

Can I use Float8 training and torch.compile for distributed LLM pretraining?▼

Yes, distributed LLM pretraining supports optional Float8 training for H100-class hardware and integrates torch.compile to provide additional speedups during large-scale model training.

Does distributed LLM pretraining support multi-node SLURM runs and resumable checkpointing?▼

Distributed LLM pretraining supports multi-node SLURM submissions and resumable distributed checkpointing, including seed checkpoint creation for pipeline parallelism to ensure production-grade workflow continuity.

What PyTorch versions are required for TorchTitan distributed training?▼

TorchTitan distributed training requires PyTorch version 2.6.0 or higher, TorchTitan 0.2.0 or higher, and TorchAO 0.5.0 or higher to properly execute 4D parallelism and checkpoint flows.

Related Skills

Explore More Software Engineering Skills→
AissenLiuAissenLiu

distributed-llm-pretraining-torchtitan

Pretrain distributed LLMs with 4D parallelism across multi-GPU clusters.

Community
Advanced
Z43LZ43L

distributed-llm-pretraining-torchtitan

Configure torchtitan TOML files for distributed LLM pretraining with 4D parallelism.

Community
Advanced
Orchestra-ResearchOrchestra-Research
11.5k

distributed-llm-pretraining-torchtitan

Pretrain large language models from scratch using PyTorch-native 4D parallelism.

Official
Advanced
AlexiosBluffMaraAlexiosBluffMara
2

distributed-llm-pretraining-torchtitan

Pretrain large language models with TorchTitan 4D parallelism on multi-GPU clusters.

Community
Advanced