skills.rest
Jensen-Yao
Jensen-YaoCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Configures and launches distributed LLM pretraining with torchtitan using 4D parallelism.

2|Updated Aug 15, 2026
Search Tags:#PyTorch#distributed training#llama#pretraining#torchtitan#float8#fsdp2
One-click install
npx skills add https://github.com/Jensen-Yao/agents-skills --skill distributed-llm-pretraining-torchtitan-jensen-yao
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/Jensen-Yao/agents-skills/tree/main/skills/torchtitan
Command: npx skills add https://github.com/Jensen-Yao/agents-skills --skill distributed-llm-pretraining-torchtitan-jensen-yao

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0, and includes references (resource) components.

What problem does it solve? Pretraining large language models from scratch across dozens or hundreds of GPUs requires correctly composing data, tensor, pipeline, and context parallelism, which is error-prone to configure manually. This Skill provides tested workflows for running PyTorch-native pretraining with torchtitan at scales from 8 to 512+ GPUs. ## Core Features & Use Cases - 4D Parallelism Configuration: Compose FSDP2, tensor parallelism, pipeline parallelism, and context parallelism for models from 8B to 405B+ parameters. - Float8 and torch.compile Acceleration: Enable Float8 training via torchao and torch.compile for up to 48% throughput gains on H100 GPUs. - Checkpoint Management: Use distributed checkpointing with async saves, seed checkpoints for pipeline parallelism, and HuggingFace/torchtune conversion. - Use Case: Pretrain a Llama 3.1 70B model on a 32-node SLURM cluster by generating the TOML config with FSDP shard degree 32 and TP degree 8, then submitting the job with sbatch. ## Quick Start Use the torchtitan skill to set up pretraining for Llama 3.1 8B on 8 GPUs with Float8 and torch.compile enabled.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain Llama 3.1 8B on multiple GPUs with torchtitan?▼

Download the HuggingFace tokenizer, create a TOML config specifying the llama3 model with 8B flavor, then launch with torchrun --nproc_per_node=8 -m torchtitan.train --job.config_file. FSDP shards the model across all GPUs by default when data_parallel_shard_degree is -1.

TorchTitan vs Megatron-LM for LLM pretraining?▼

TorchTitan is PyTorch-native with composable 4D parallelism and no third-party dependencies, interoperable with torchtune and HuggingFace checkpoints. Megatron-LM targets maximum performance for NVIDIA-only deployments but involves a heavier framework.

Does torchtitan support Float8 training on H100 GPUs?▼

Yes, torchtitan supports Float8 training on H100 or newer GPUs via torchao, providing 30-50% speedups. Enable it with the quantize.linear.float8 model converter plus torch.compile, using tensorwise or rowwise scaling recipes.

How do I resume torchtitan training from a checkpoint?▼

Training auto-resumes from the latest checkpoint in the configured folder when checkpoint.enable is true. To resume from a specific step, set checkpoint.load_step to that step number in the TOML config.

Why does torchtitan run out of memory on large models?▼

OOM occurs when activation memory exceeds GPU capacity. Enable full activation checkpointing, reduce local_batch_size to 1, or use gradient accumulation with a larger global_batch_size to fit the model.

Can I convert torchtitan checkpoints to HuggingFace format?▼

Yes, torchtitan supports saving directly in HuggingFace format with last_save_in_hf, or offline conversion using the convert_to_hf.py and convert_from_hf.py scripts. Model-only checkpoints also load directly into torchtune for fine-tuning.

Related Skills

Explore More Software Engineering Skills→
JamesFincherJamesFincher

distributed-llm-pretraining-torchtitan

Configure TorchTitan for distributed LLM pretraining with 4D parallelism.

Community
Advanced
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
GonglitianGonglitian
2

gpu-train-monitor

Monitors GPU health and summarizes training progress from live system data.

Community
Advanced
BagelHoleBagelHole
46

llm-fine-tuning

Set up QLoRA, LoRA, and full fine-tuning infrastructure with Hugging Face TRL and Axolotl.

Community
Advanced