skills.rest
Supporter09
Supporter09COMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Scale LLM pretraining with PyTorch-native distributed training and 4D parallelism.

Updated May 4, 2026
Search Tags:#checkpointing#distributed training#torch.compile#torchtitan#float8#fsdp2#llm pretraining
One-click install
npx skills add https://github.com/Supporter09/Face_Anti_Spoofing_Biometric --skill distributed-llm-pretraining-torchtitan-supporter09
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/Supporter09/Face_Anti_Spoofing_Biometric/tree/main/.claude/skills/torchtitan
Command: npx skills add https://github.com/Supporter09/Face_Anti_Spoofing_Biometric --skill distributed-llm-pretraining-torchtitan-supporter09

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

TorchTitan reduces the complexity of launching and maintaining large language model pretraining by providing a PyTorch-native stack for distributed training, checkpointing, and scalable configuration.

Core Features & Use Cases

  • 4D parallelism with FSDP2, tensor parallelism, pipeline parallelism, and context parallelism for training from 8 to 512+ GPUs.
  • Float8 and torch.compile support for higher throughput on H100 and newer hardware.
  • Distributed checkpointing, HuggingFace conversion, and seed checkpoint workflows for reliable resume and multi-parallelism changes.
  • Custom model integration through TrainSpec patterns and reusable training configs.
  • Use cases include pretraining Llama 3.1, DeepSeek V3, GPT-OSS, Qwen 3, or custom transformer models at scale.

Quick Start

Install TorchTitan and launch a training run with the provided Llama configuration on your target GPU cluster.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I pretrain LLMs across 512+ GPUs using PyTorch?▼

To pretrain LLMs at scale, this Skill applies 4D parallelism combining FSDP2, tensor parallelism, pipeline parallelism, and context parallelism. It scales PyTorch-native distributed training from 8 to 512+ GPUs for models like Llama 3.1 and DeepSeek V3.

How does Float8 tuning work with torch.compile for distributed training?▼

Float8 tuning works with torch.compile to increase training throughput on H100 and newer hardware. This Skill integrates both features natively to optimize large language model pretraining performance on supported GPU clusters.

Can I resume distributed training with a different parallelism configuration?▼

Yes, you can resume distributed training with a different parallelism configuration. This Skill uses distributed checkpointing and seed checkpoint workflows to support reliable resume and multi-parallelism changes across your training runs.

Does this Skill support checkpoint conversion for HuggingFace models?▼

Yes, this Skill supports HuggingFace conversion for distributed checkpoints. It provides PyTorch Distributed Checkpoint interoperability to convert and load pretraining states into HuggingFace formats.

What is the best way to pretrain custom transformer models at scale?▼

The best way to pretrain custom transformer models is through TrainSpec patterns and reusable training configs. This Skill allows custom model integration for large-scale PyTorch-native distributed pretraining workflows.

Related Skills

Explore More Software Engineering Skills→
jleechanorgjleechanorg

distributed-llm-pretraining-torchtitan

Configure distributed LLM pretraining with PyTorch 4D parallelism across GPUs.

Official
Advanced
ruiyangruiyiruiyangruiyi

axolotl

Configure and validate Axolotl YAML workflows for LLM fine-tuning with LoRA/QLoRA and DPO/GRPO.

Community
Advanced
MesferAliMesferAli

distributed-llm-pretraining-torchtitan

Configure TorchTitan 4D parallelism for distributed LLM pretraining on H100 GPUs.

Community
Advanced
thisismynewfmail-uithisismynewfmail-ui

distributed-llm-pretraining-torchtitan

Pretrain large language models with distributed TorchTitan pipelines using 4D parallelism.

Community
Advanced