skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Pretrain large language models with PyTorch-native 4D parallelism using TorchTitan.

Updated Jun 5, 2026
Search Tags:#pytorch#distributed training#fsdp#tensor parallelism#torchtitan#float8#llm pretraining
One-click install
npx skills add https://github.com/xu1713/openhorse --skill distributed-llm-pretraining-torchtitan-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/xu1713/openhorse --skill distributed-llm-pretraining-torchtitan-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, torchtitan, torchao, and includes references (resource) components.

What problem does it solve? Pretraining large language models from scratch requires coordinating complex distributed training strategies across hundreds of GPUs, and configuring FSDP, tensor parallelism, pipeline parallelism, and Float8 quantization correctly is error-prone and time-consuming. ## Core Features & Use Cases - 4D Parallelism Configuration: Compose FSDP2, tensor parallelism, pipeline parallelism, and context parallelism to scale training from 8 to 512+ GPUs for models like Llama 3.1 8B through 405B. - Float8 and torch.compile Optimization: Enable Float8 training with torchao and torch.compile for up to 48% throughput gains on H100 GPUs. - Checkpoint Management: Use distributed checkpointing with async saves, HuggingFace conversion, and seed checkpoints for pipeline parallelism. - Use Case: A research team wants to pretrain a Llama 3.1 70B model on 256 GPUs across 32 SLURM-managed nodes. This Skill provides the TOML configuration, SLURM script, and parallelism settings to launch and monitor the run. ## Quick Start Ask the AI to configure and launch a TorchTitan pretraining run for Llama 3.1 8B on 8 GPUs with a TOML config file.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain Llama 3.1 8B on multiple GPUs with TorchTitan?▼

Download the tokenizer with the download_hf_assets.py script, create a TOML config specifying the llama3 model and 8B flavor, then launch with torchrun --nproc_per_node=8 -m torchtitan.train --job.config_file your_config.toml. FSDP sharding across all GPUs is the default.

TorchTitan vs Megatron-LM for LLM pretraining?▼

TorchTitan is PyTorch-native with composable 4D parallelism and no third-party dependencies, making it easier to modify and integrate with torchtune and HuggingFace. Megatron-LM targets maximum performance for NVIDIA-only deployments.

Does TorchTitan support Float8 training on H100 GPUs?▼

Yes, TorchTitan supports Float8 training on H100 GPUs via torchao with tensorwise or rowwise scaling. Combined with torch.compile, benchmarks show up to 48% throughput improvement over the FSDP baseline on Llama 8B.

How do I convert TorchTitan checkpoints to HuggingFace format?▼

Use the convert_to_hf.py script in scripts/checkpoint_conversion with the input directory, output directory, model name, and flavor. You can also set last_save_in_hf = true in the checkpoint config to save directly in HuggingFace format during training.

Why does TorchTitan run out of memory on large models?▼

Out-of-memory errors occur when activation memory exceeds GPU capacity. Enable full activation checkpointing, reduce local_batch_size, or use gradient accumulation with a larger global_batch_size in the TOML config.

When should I not use TorchTitan for training?▼

Avoid TorchTitan for fine-tuning tasks, where Axolotl, TRL, or torchtune are better suited. It is designed for pretraining from scratch at 8B to 405B+ scale, not parameter-efficient fine-tuning or inference serving.

Related Skills

Explore More Software Engineering Skills→
kaminocorpkaminocorp
1

nemo-curator

Curate LLM training datasets with GPU-accelerated deduplication, filtering, and PII redaction.

Official
Advanced
itallstartedwithaideaitallstartedwithaidea
31

ml-model-integration

Select and deploy HuggingFace Hub models with task-first filtering and evaluation.

Community
Advanced
henrycashe26henrycashe26

ml-training-monitor

Diagnose machine learning training runs by interpreting loss curves and validation metrics.

Community
Advanced
arsityarsity

huggingface-accelerate

Run distributed PyTorch training across DDP, DeepSpeed, FSDP, and Megatron backends.

Community
Advanced