skills.rest
Plaidmustache
PlaidmustacheCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Pretrain large language models with 4D parallelism across 8 to 512+ GPUs.

Updated May 4, 2026
Search Tags:#distributed-training#tensor-parallel#pipeline-parallel#torchtitan#float8#fsdp2#llm-pretraining
One-click install
npx skills add https://github.com/Plaidmustache/hermes-nulab --skill distributed-llm-pretraining-torchtitan-plaidmustache
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/Plaidmustache/hermes-nulab/tree/main/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/Plaidmustache/hermes-nulab --skill distributed-llm-pretraining-torchtitan-plaidmustache

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provides PyTorch-native distributed LLM pretraining using TorchTitan's 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.

Core Features & Use Cases

  • 4D parallelism (FSDP2, TP, PP, CP) for scalable LLM pretraining across many GPUs.
  • PyTorch-native training with interoperability; minimizes external dependencies.
  • Float8 support with torch.compile and distributed checkpointing for high-throughput training.
  • Suitable for pretraining Llama 3.1, DeepSeek V3, or custom models from 8B to 512B+ parameters.

Quick Start

Begin a 4D-parallel pretraining run on a multi-GPU cluster using TorchTitan to scale LLMs from 8B to 512B+ parameters.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain large language models at scale using 4D parallelism in PyTorch?▼

Pretrain large language models at scale using PyTorch-native 4D parallelism, combining FSDP2, tensor, pipeline, and context parallelism to efficiently train architectures from 8B to 512B+ parameters.

Can I use Float8 and torch.compile to speed up distributed LLM pretraining?▼

Float8 support and torch.compile are integrated to accelerate distributed LLM pretraining, enabling high-throughput training across 8 to 512+ GPUs while minimizing external dependencies.

What is the best way to scale Llama 3.1 or DeepSeek V3 pretraining across a multi-GPU cluster?▼

Scale Llama 3.1 or DeepSeek V3 pretraining across a multi-GPU cluster using TorchTitan's 4D parallelism, which supports distributed checkpointing and PyTorch-native interoperability.

Does this distributed training approach work with HuggingFace and torchtune?▼

This PyTorch-native distributed training approach ensures interoperability with standard tools like HuggingFace and torchtune, minimizing external dependencies while maintaining workflow compatibility.

When do I need FSDP2 and distributed checkpointing for LLM pretraining?▼

Use FSDP2 and distributed checkpointing for LLM pretraining when scaling models from 8B to 512B+ parameters across 8 to 512+ GPUs to ensure high-throughput training and reliable state recovery.

Related Skills

Explore More Software Engineering Skills→
JuanMS20JuanMS20

huggingface-accelerate

Configure distributed PyTorch training across multi-GPU and multi-node devices.

Community
Advanced
ovachieverovachiever
52

ray-data

Process distributed ML data pipelines with Ray Data across CPU/GPU clusters.

Community
Advanced
heysuhasheysuhas
2

axolotl

Manage YAML-based configuration for fine-tuning LLMs with PyTorch and HuggingFace.

Community
Advanced
ovachieverovachiever
52

training-llms-megatron

Train large language models with Megatron-Core using configurable 3D and 4D parallelism.

Community
Advanced