skills.rest
ruiyangruiyi
ruiyangruiyiCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Configure TorchTitan distributed LLM pretraining with 4D parallelism and Float8.

Updated May 26, 2026
Search Tags:#checkpointing#tensor-parallelism#pipeline-parallelism#float8#fsdp2#distributed-llm-pretraining#torch-titan
One-click install
npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill distributed-llm-pretraining-torchtitan-ruiyangruiyi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/ruiyangruiyi/hermes-agent/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill distributed-llm-pretraining-torchtitan-ruiyangruiyi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, torchtitan, torchao, and includes references (resource) components.

What problem does it solve?

Distributed LLM pretraining at scale is hard to configure and optimize, especially when you need high throughput across many GPUs while keeping memory usage manageable and checkpoints interoperable.

Core Features & Use Cases

  • PyTorch-native distributed pretraining with TorchTitan: Set up composable 4D parallelism using FSDP2, Tensor Parallel (TP), Pipeline Parallel (PP), and Context Parallel (CP) for efficient training from scratch.
  • Scale and performance oriented workflows: Train models like Llama 3.1 and DeepSeek V3 from 8 to 512+ GPUs, including distributed checkpointing and torch.compile integration.
  • H100 Float8 optimization: Enable Float8 training (via torchao) to improve throughput when GEMMs are large enough to benefit from FP8 tensor core acceleration.

Quick Start

Instruct the AI to generate a TorchTitan TOML config for an 8-GPU Llama 3.1 pretraining run using Float8 and torch.compile, then provide the exact command to launch training on that node.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I set up distributed LLM pretraining with 4D parallelism in PyTorch?▼

Distributed LLM pretraining with 4D parallelism is configured using TorchTitan to compose FSDP2, Tensor Parallel, Pipeline Parallel, and Context Parallel for efficient large-scale training from scratch.

Can I use Float8 training to speed up LLM pretraining on H100 GPUs?▼

Float8 training acceleration on H100 GPUs is supported via torchao, which improves throughput by leveraging FP8 tensor cores when GEMMs are large enough to benefit from the acceleration.

What is the best way to resume distributed checkpoints for Llama 3.1 pretraining?▼

Resuming distributed checkpoints for Llama 3.1 pretraining is handled by interoperable checkpointing compatible with both TorchTitan and HuggingFace workflows, ensuring seamless training continuation.

Does TorchTitan support SLURM multi-node pretraining for models up to 405B parameters?▼

SLURM multi-node pretraining for models from 8B to 405B+ parameters is fully supported by TorchTitan, allowing scalable training runs across 8 to 512+ GPUs with FSDP2 sharding.

How do I generate a TorchTitan TOML config and launch an 8-GPU pretraining run?▼

Generating a TorchTitan TOML config for an 8-GPU Llama 3.1 pretraining run with Float8 and torch.compile is done by instructing the AI, which then provides the exact command to launch training on that node.

Related Skills

Explore More Software Engineering Skills→
Orchestra-ResearchOrchestra-Research
11.5k

pytorch-fsdp2

Integrate PyTorch FSDP2 with DTensor sharding and DCP checkpointing into training scripts.

Official
Advanced
Devsoul2026Devsoul2026
150

distributed-llm-pretraining-torchtitan

Optimize scalable LLM pretraining with PyTorch-native 4D parallelism and distributed checkpointing.

Community
Advanced
qcmuuqcmuu

distributed-llm-pretraining-torchtitan

Configure PyTorch-native distributed LLM pretraining with torchtitan 4D parallelism.

Community
Advanced
ovachieverovachiever
52

training-llms-megatron

Train large language models with Megatron-Core using configurable 3D and 4D parallelism.

Community
Advanced