skills.rest
attentiondotnet
attentiondotnetCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

distributed-llm-pretraining-torchtitan

Distribute LLM pretraining with PyTorch-native 4D parallelism across multi-GPU clusters.

Updated Mar 30, 2026
Search Tags:#checkpointing#distributed-training#tensor-parallel#torchtitan#float8#fsdp2#llm-pretraining
One-click install
npx skills add https://github.com/attentiondotnet/hermes-agent --skill distributed-llm-pretraining-torchtitan-attentiondotnet
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/attentiondotnet/hermes-agent/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/attentiondotnet/hermes-agent --skill distributed-llm-pretraining-torchtitan-attentiondotnet

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TorchTitan enables PyTorch-native, large-scale LLM pretraining by providing a composable 4D parallelism framework, allowing efficient scaling from small to massive models across multi-GPU setups.

Core Features & Use Cases

  • 4D parallelism (FSDP2, TP, PP, CP) for scalable pretraining
  • Float8 training support on H100/Blackwell with optional compile for speedups
  • Interoperable with HuggingFace checkpoints and torchtune, plus distributed checkpointing
  • Workflow templates for single-node to multi-node training and seed checkpoint creation

Quick Start

Install TorchTitan, configure a 4D parallelism setup, and launch your pretraining job on your LLM.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I scale LLM pretraining across a multi-GPU cluster using PyTorch?▼

Scale LLM pretraining across multi-GPU clusters using PyTorch-native 4D parallelism. This framework composes FSDP2, Tensor Parallelism, Pipeline Parallelism, and Context Parallelism to efficiently distribute workloads from 8 to 512+ GPUs.

Can I use Float8 training for LLM pretraining on H100 GPUs?▼

Yes, Float8 training for LLM pretraining is supported on H100 and Blackwell GPUs. It includes optional torch compile integration to maximize hardware utilization and deliver additional training speedups.

How do I manage distributed checkpointing for large-scale pretraining?▼

Manage distributed checkpointing using interoperable formats compatible with HuggingFace and torchtune. The workflow supports robust checkpoint saving, loading, and seed checkpoint creation to ensure pretraining reliability.

What is the best way to orchestrate 4D parallelism for LLM pretraining?▼

Orchestrate 4D parallelism for LLM pretraining using composable PyTorch-native primitives. This approach integrates FSDP2, TP, PP, and CP with compile-ready model paths to enable end-to-end scalable training workflows.

Does this distributed training framework work with HuggingFace checkpoints?▼

Yes, the distributed training framework works directly with HuggingFace checkpoints. It ensures interoperability with HuggingFace and torchtune, allowing seamless loading and saving of model states during pretraining.

Related Skills

Explore More Software Engineering Skills→
tadod12tadod12

torchforge-rl-training

Train RL experiments with torchforge separating infrastructure from algorithms in PyTorch.

Community
Advanced
Clay-HHKClay-HHK
2

llama-factory

Fine-tune LLMs via WebUI/CLI with LoRA/QLoRA workflows and adapter merging.

Community
Advanced
Supporter09Supporter09

distributed-llm-pretraining-torchtitan

Scale LLM pretraining with PyTorch-native distributed training and 4D parallelism.

Community
Advanced
qcmuuqcmuu

training-llms-megatron

Configure Megatron-Core distributed training for large language models.

Community
Advanced