skills.rest
Chensihakniroth
ChensihaknirothCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Pretrain large language models at scale using PyTorch-native 4D parallelism with torchtitan.

Updated Jun 7, 2026
Search Tags:#PyTorch#distributed training#fsdp#tensor parallelism#torchtitan#float8#llm pretraining
One-click install
npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill distributed-llm-pretraining-torchtitan-chensihakniroth
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/Chensihakniroth/ANAKOT-AGENT/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/Chensihakniroth/ANAKOT-AGENT --skill distributed-llm-pretraining-torchtitan-chensihakniroth

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0, and includes references (resource) components.

What problem does it solve? Pretraining large language models from scratch requires coordinating complex distributed training strategies across hundreds of GPUs, and misconfiguring parallelism, checkpointing, or memory settings leads to out-of-memory failures and wasted compute. This Skill provides structured workflows for running PyTorch-native distributed pretraining with torchtitan, covering FSDP2, tensor, pipeline, and context parallelism. ## Core Features & Use Cases - 4D Parallelism Configuration: Compose FSDP2, tensor parallelism, pipeline parallelism, and context parallelism to train models from 8B to 405B+ parameters across 8 to 512+ GPUs. - Float8 and torch.compile Acceleration: Enable Float8 training with torchao on H100 GPUs for 30-50% speedups, with layer filtering and rowwise scaling recipes. - Distributed Checkpointing: Use PyTorch Distributed Checkpoint for fault-tolerant saves, async checkpointing, seed checkpoints for pipeline parallelism, and HuggingFace format conversion. - Use Case: A research team wants to pretrain a Llama 3.1 70B model on 256 GPUs across 32 SLURM-managed nodes. Use this Skill to configure FSDP plus tensor parallelism, generate the SLURM submission script, and set up automatic checkpoint resumption. ## Quick Start Ask the agent to configure and launch a torchtitan pretraining run for Llama 3.1 8B on 8 GPUs with a TOML config file and checkpointing enabled.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain Llama 3.1 on multiple GPUs with torchtitan?▼

Download the tokenizer with the download_hf_assets.py script, create a TOML config specifying model flavor and parallelism degrees, then launch with torchrun or the run_train.sh wrapper. For 8 GPUs on one node, set data_parallel_shard_degree to -1 to use all GPUs for FSDP.

Torchtitan vs Megatron-LM for LLM pretraining?▼

Torchtitan is PyTorch-native with composable 4D parallelism and no third-party dependencies, making it easier to modify and interoperable with torchtune and HuggingFace. Megatron-LM targets maximum performance for NVIDIA-only deployments but with a heavier framework.

Does torchtitan support Float8 training on H100 GPUs?▼

Yes, torchtitan supports Float8 training via torchao on H100 or newer GPUs with FP8 tensor cores. Enable it with the quantize.linear.float8 model converter plus torch.compile, which benchmarks show delivers up to 48% throughput improvement over FSDP baselines.

Why does torchtitan run out of memory on large models?▼

Out-of-memory errors occur when activation memory exceeds GPU capacity. Enable full activation checkpointing instead of selective mode, reduce local_batch_size, or use gradient accumulation with a larger global_batch_size to lower peak memory usage.

How do I convert torchtitan checkpoints to HuggingFace format?▼

Use the convert_to_hf.py script in scripts/checkpoint_conversion with the input directory, output directory, model name, and flavor. Alternatively, set last_save_in_hf to true in the checkpoint config to save directly in HuggingFace format during training.

When should I not use torchtitan for training?▼

Avoid torchtitan for fine-tuning tasks, where Axolotl, TRL, or torchtune are better suited, and for inference workloads where DeepSpeed offers broader support. It is designed specifically for pretraining LLMs from scratch at scale.

Related Skills

Explore More Software Engineering Skills→
GregsGreyCodeGregsGreyCode
6

huggingface-tokenizers

Integrates HuggingFace's fast tokenizers for efficient NLP text processing in Python projects.

Community
Advanced
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
yo-stevenyo-steven

mlm-bridge-training

Correlate Megatron-LM and Megatron Bridge training runs by matching configs and diagnosing loss divergence.

Community
Advanced
arsityarsity

huggingface-accelerate

Run distributed PyTorch training across DDP, DeepSpeed, FSDP, and Megatron backends.

Community
Advanced