skills.rest
MLT-OSS
MLT-OSSOFFICIAL
Explore all skills by this author →
WebsiteGitHub

distributed-llm-pretraining-torchtitan

Configures and launches distributed LLM pretraining with TorchTitan using 4D parallelism.

14|5|Updated Apr 9, 2026
Search Tags:#pytorch#distributed training#fsdp#tensor parallelism#torchtitan#float8#llm pretraining
One-click install
npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill distributed-llm-pretraining-torchtitan-mlt-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill distributed-llm-pretraining-torchtitan-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch>=2.6.0, torchtitan>=0.2.0, torchao>=0.5.0, and includes references (resource) components.

What problem does it solve? Pretraining large language models from scratch across dozens or hundreds of GPUs requires correctly configuring data, tensor, pipeline, and context parallelism, checkpointing, and mixed precision—an error-prone process that this Skill turns into guided, reproducible workflows. ## Core Features & Use Cases - 4D Parallelism Configuration: Compose FSDP2, tensor, pipeline, and context parallelism for models from 8B to 405B+ parameters across 8 to 512+ GPUs. - Float8 and torch.compile Training: Enable Float8 quantization with torchao and torch.compile for up to 48% throughput gains on H100 GPUs. - Checkpoint Management: Use distributed checkpointing with async saves, seed checkpoints for pipeline parallelism, and HuggingFace/torchtune conversion. - Use Case: Pretrain Llama 3.1 70B on 256 GPUs by generating a TOML config with FSDP shard degree 32 and TP degree 8, then submitting a SLURM job with automatic checkpoint resume. ## Quick Start Ask the agent to set up a TorchTitan training run for Llama 3.1 8B on 8 GPUs, including tokenizer download, TOML config, and launch command.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain Llama 3.1 8B on multiple GPUs with TorchTitan?▼

Download the tokenizer with the download_hf_assets.py script, create a TOML config specifying the llama3 model and 8B flavor, then launch with torchrun --nproc_per_node=8 -m torchtitan.train --job.config_file. FSDP shards the model across all GPUs by default.

TorchTitan vs Megatron-LM vs DeepSpeed for LLM pretraining?▼

TorchTitan is PyTorch-native with composable 4D parallelism and no third-party framework dependencies. Megatron-LM targets maximum NVIDIA-only performance, while DeepSpeed offers a broader ZeRO optimization ecosystem and inference support.

Does TorchTitan support Float8 training on H100 GPUs?▼

Yes, TorchTitan supports Float8 training on H100s via torchao with tensorwise or rowwise scaling. Combined with torch.compile, benchmarks show up to 48% throughput improvement over the FSDP baseline on Llama 8B.

How do I convert TorchTitan checkpoints to HuggingFace format?▼

Use the convert_to_hf.py script in scripts/checkpoint_conversion with the model name and flavor, or set last_save_in_hf = true in the checkpoint config to save directly in HuggingFace format during training.

Why does TorchTitan run out of memory on large models?▼

OOM typically occurs when activation memory exceeds GPU capacity. Enable full activation checkpointing, reduce local_batch_size, or use gradient accumulation with a larger global_batch_size to fit the model.

When should I not use TorchTitan for training?▼

Avoid TorchTitan for fine-tuning workflows, where Axolotl or TRL are better suited, and for small-scale educational training where LitGPT is simpler. It is designed specifically for large-scale pretraining from scratch.

Related Skills

Explore More Software Engineering Skills→
kaminocorpkaminocorp
1

nemo-curator

Curate LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

Official
Advanced
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
yo-stevenyo-steven

mlm-bridge-training

Correlate Megatron-LM and Megatron Bridge training runs by matching configs and diagnosing loss divergence.

Community
Advanced
BagelHoleBagelHole
46

llm-fine-tuning

Set up QLoRA, LoRA, and full fine-tuning infrastructure with Hugging Face TRL and Axolotl.

Community
Advanced