Pretrain large language models using PyTorch native 4D parallelism.
npx skills add https://github.com/NousResearch/hermes-agent --skill torchtitan
Please help me install this Agent Skill. Skill: torchtitan Source: https://github.com/NousResearch/hermes-agent/tree/main/optional-skills/mlops/torchtitan Command: npx skills add https://github.com/NousResearch/hermes-agent --skill torchtitan
Pretraining large language models from scratch requires coordinating thousands of GPUs across multiple parallelism strategies, managing memory-efficient sharding, and integrating low-precision formats like Float8. This Skill provides structured workflows for configuring and launching TorchTitan training runs at scale.
Use the torchtitan skill to configure and launch a Llama 3.1 8B pretraining run on 8 GPUs with FSDP2 and Float8 enabled.
High-intent search queries and answers about installing and using this skill.
Use TorchTitan with FSDP2 for sharding, Tensor Parallel for intra-node splitting, and Pipeline Parallel for cross-node stages. Launch via torchrun or SLURM with the run_train.sh script and a registered config name from the Python config registry.
FSDP2 uses DTensor-based sharding without the FlatParameter abstraction, achieving 7% lower peak memory and better composability. It supports meta-device initialization and communication-free sharded state dicts compared to FSDP1.
Yes, TorchTitan integrates torchao Float8LinearConverter with rowwise or tensorwise scaling recipes. Float8 requires torch.compile and benefits GEMMs with K,N dimensions greater than 4096, delivering 30-50% speedups on H100 hardware.
Pipeline parallelism splits model layers across stages, requiring identical weight initialization across all ranks. A seed checkpoint created on a single GPU ensures consistent initialization before sharding across the pipeline.
Use Megatron-LM when maximum throughput on NVIDIA-only deployments is the priority. TorchTitan is preferable for PyTorch-native workflows requiring composable parallelism, HuggingFace interoperability, and broader hardware support.
JamesFincherConfigure TorchTitan for distributed LLM pretraining with 4D parallelism.
ScientiaCapitalExport fine-tuned models to GGUF and deploy with vLLM or Ollama.
NVIDIAIntegrate nvidia-resiliency-ext fault tolerance into Megatron Bridge training workflows.
Orchestra-ResearchTrains large-scale language models using NVIDIA Megatron-Core with advanced parallelism strategies.
