skills.rest
loteiron
loteironCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

torchtitan

Pretrain large language models with PyTorch 4D parallelism and Float8 training.

Updated Sep 10, 2026
Search Tags:#PyTorch#distributed training#torchtitan#4d parallelism#float8#fsdp2#llm pretraining
One-click install
npx skills add https://github.com/loteiron/ZeusAgent --skill torchtitan-loteiron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: torchtitan
Source: https://github.com/loteiron/ZeusAgent/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/loteiron/ZeusAgent --skill torchtitan-loteiron

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0, and includes references (resource) components.

What problem does it solve? Pretraining LLMs from scratch at 8B to 405B+ scale requires coordinating FSDP2, tensor, pipeline, and context parallelism across hundreds of GPUs, plus checkpointing, Float8 quantization, and SLURM orchestration, which is complex to configure correctly. ## Core Features & Use Cases - Composable 4D Parallelism: Configure FSDP2, tensor parallel, pipeline parallel, and context parallel degrees for models from 8B to 405B on single nodes or 512-GPU clusters. - Float8 and Compile Acceleration: Enable Float8 rowwise scaling with torch.compile for up to 48% throughput gains on H100 GPUs. - Checkpoint Management: Use PyTorch Distributed Checkpoint for async saves, seed checkpoints for pipeline parallelism, and HuggingFace/torchtune conversion. - Use Case: Pretrain Llama 3.1 8B on 8 GPUs by downloading the tokenizer, selecting the llama3_8b config from the registry, and launching with run_train.sh while monitoring TensorBoard logs. ## Quick Start Ask the agent to pretrain a Llama 3.1 8B model on 8 GPUs using TorchTitan with the llama3_8b config and checkpointing every 500 steps.

Frequently Asked Questions about torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain Llama 3.1 8B with TorchTitan?▼

Download the Llama 3.1 tokenizer with the download_hf_assets.py script, then launch training with MODULE=llama3 CONFIG=llama3_8b ./run_train.sh on 8 GPUs. Configs are selected by name from the Python config registry, and fields can be overridden via command-line flags like --optimizer.lr.

TorchTitan vs Megatron-LM vs DeepSpeed for LLM pretraining?▼

TorchTitan is PyTorch-native with composable FSDP2, TP, PP, and CP parallelism and no third-party dependencies. Megatron-LM targets maximum NVIDIA-only performance, DeepSpeed offers a broader ZeRO ecosystem, and Axolotl or TRL suit fine-tuning rather than pretraining.

Does TorchTitan support Float8 training on H100 GPUs?▼

Yes, TorchTitan supports Float8 via torchao with tensorwise and rowwise scaling recipes on H100 or newer GPUs. Enable it through Float8LinearConverter in the config registry with torch.compile, yielding up to 48% throughput improvement over FSDP baselines.

How do I run multi-node TorchTitan training with SLURM?▼

Create a SLURM script with srun torchrun specifying --nnodes, --nproc_per_node, and c10d rendezvous settings, then submit with sbatch. Configure parallelism degrees such as data_parallel_shard_degree and tensor_parallel_degree to match your cluster topology.

Why does TorchTitan run out of memory on large models?▼

Out of memory occurs when activation storage exceeds GPU capacity. Enable full activation checkpointing, reduce local_batch_size to 1, or use gradient accumulation with a larger global_batch_size to fit the model.

How do I convert TorchTitan checkpoints to HuggingFace format?▼

Use the convert_to_hf.py script with the input directory, output directory, model name, and flavor, or set last_save_in_hf = true during training. DCP sharded checkpoints can also be merged into a single .pt file with torch.distributed.checkpoint format utilities.

Related Skills

Explore More Software Engineering Skills→
GregsGreyCodeGregsGreyCode
6

huggingface-tokenizers

Integrates HuggingFace's fast tokenizers for efficient NLP text processing in Python projects.

Community
Advanced
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
GonglitianGonglitian
2

gpu-train-monitor

Monitors GPU health and summarizes training progress from live system data.

Community
Advanced
BagelHoleBagelHole
46

llm-fine-tuning

Set up QLoRA, LoRA, and full fine-tuning infrastructure with Hugging Face TRL and Axolotl.

Community
Advanced