skills.rest
m0at
m0atCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Orchestrate 4D parallelism for large-scale LLM pretraining with TorchTitan.

1|Updated Apr 4, 2026
Search Tags:#distributed-training#tensor-parallel#torchtitan#float8#fsdp2#llm-pretraining
One-click install
npx skills add https://github.com/m0at/claudemd --skill distributed-llm-pretraining-torchtitan-m0at
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/m0at/claudemd/tree/main/skills/torchtitan
Command: npx skills add https://github.com/m0at/claudemd --skill distributed-llm-pretraining-torchtitan-m0at

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TorchTitan enables PyTorch-native distributed pretraining of large language models using composable 4D parallelism across multi-GPU clusters.

Core Features & Use Cases

  • 4D parallelism across FSDP2, tensor parallelism (TP), pipeline parallelism (PP), and context parallelism (CP) to scale pretraining.
  • Float8 training on H100 with selective filtering and compile-stage acceleration for speedups.
  • Checkpoint interoperability with HuggingFace and torchtune to save, resume, and convert checkpoints.
  • Multi-node workflows including 8 to 512+ GPU clusters and SLURM-based launches for large-scale pretraining.

Quick Start

Launch a pretraining job on 8 GPUs using a ready config and the training script.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I orchestrate 4D parallelism for LLM pretraining across multiple GPUs?▼

You can orchestrate 4D parallelism for LLM pretraining by using TorchTitan to composable scale FSDP2, tensor parallelism, pipeline parallelism, and context parallelism across multi-GPU clusters.

Can I use Float8 training on H100 clusters to speed up large-scale LLM pretraining?▼

Yes, you can use Float8 training on H100 clusters to accelerate large-scale LLM pretraining, utilizing selective filtering and compile-stage acceleration for optimized speedups.

How do I launch distributed LLM pretraining jobs on 8 to 512+ GPU clusters?▼

You can launch distributed LLM pretraining jobs on 8 to 512+ GPU clusters using TorchTitan's multi-node workflows, which support ready configs and SLURM-based launches for large-scale execution.

Does TorchTitan support checkpoint interoperability with HuggingFace for pretraining workflows?▼

Yes, TorchTitan supports checkpoint interoperability with HuggingFace and torchtune, allowing you to save, resume, and convert distributed checkpoints seamlessly during LLM pretraining.

What models are supported for PyTorch-native distributed pretraining?▼

PyTorch-native distributed pretraining supports Llama 3.1, DeepSeek V3, and custom models, enabling you to optimize large-scale training workflows across multi-node GPU clusters.

Related Skills

Explore More Software Engineering Skills→
Z43LZ43L

distributed-llm-pretraining-torchtitan

Configure torchtitan TOML files for distributed LLM pretraining with 4D parallelism.

Community
Advanced
frankxaifrankxai
2

HuggingFace Model Trainer

Fine-tune large language models with HuggingFace TRL, Transformers, and PEFT.

Community
Advanced
Orchestra-ResearchOrchestra-Research
11.5k

distributed-llm-pretraining-torchtitan

Pretrain large language models from scratch using PyTorch-native 4D parallelism.

Official
Advanced
MarbleSodasMarbleSodas

distributed-llm-pretraining-torchtitan

Configure torchtitan TOML jobs for 4D parallel LLM pretraining on GPUs.

Community
Advanced