skills.rest
ricable
ricableCOMMUNITY
Explore all skills by this author →
GitHub

distributed-llm-pretraining-torchtitan

Automates Torch-based pretraining runs with 4D parallelism and Float8 optimization for Llama, DeepSeek, and custom architectures.

Updated Feb 15, 2026
Search Tags:#checkpointing#distributed training#tensor parallelism#torchtitan#float8#fsdp2#llm pretraining
One-click install
npx skills add https://github.com/ricable/mcai --skill distributed-llm-pretraining-torchtitan-ricable
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/ricable/mcai/tree/main/.agents/skills/distributed-llm-pretraining-torchtitan
Command: npx skills add https://github.com/ricable/mcai --skill distributed-llm-pretraining-torchtitan-ricable

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the complexity of configuring and operating large-scale TorchTitan pretraining runs, helping teams launch stable distributed training without stitching together parallelism, checkpointing, and model-specific setup by hand.

Core Features & Use Cases

  • Composable 4D parallelism across FSDP2, tensor parallelism, pipeline parallelism, and context parallelism for scaling from 8 GPUs to 512+ GPUs.
  • Float8 training support for H100 and newer hardware, with torch.compile integration for higher throughput.
  • Fault-tolerant distributed checkpointing, HuggingFace import and export, and seed checkpoint creation for pipeline parallelism.
  • Custom model integration guidance for adding new architectures, train specs, and state-dict adapters.
  • Use case: pretrain Llama 3.1, DeepSeek V3, or a custom model on single-node or SLURM-managed multi-node clusters, then resume training from the latest checkpoint or convert checkpoints for fine-tuning.

Quick Start

Use this skill to plan or explain a TorchTitan pretraining run for your model, GPU count, and checkpointing setup.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
Does TorchTitan support DeepSeek V3 and Qwen pretraining alongside torch.compile?▼

TorchTitan supports pretraining for DeepSeek V3, Qwen, GPT-OSS, and Flux models. These architectures integrate with torch.compile and float8 optimization for enhanced training throughput.

Can I resume multi-node pretraining from a distributed checkpoint on SLURM?▼

You can resume multi-node pretraining on SLURM-managed clusters using fault-tolerant distributed checkpointing. The workflow supports saving and loading checkpoints to recover training without losing progress.

How do I export distributed checkpoints to HuggingFace format for fine-tuning?▼

Export distributed checkpoints to HuggingFace format using the built-in conversion support. This allows seamless transition from large-scale pretraining to downstream fine-tuning workflows.

What is the best way to add a custom model architecture to TorchTitan pretraining?▼

Add custom model architectures by following the integration guidance for new train specs and state-dict adapters. This enables your specific model to utilize the 4D parallelism and checkpointing features.

Does TorchTitan support DeepSeek V3 and Qwen pretraining alongside torch.compile?▼

TorchTitan supports pretraining for DeepSeek V3, Qwen, GPT-OSS, and Flux models. These architectures integrate with torch.compile and float8 optimization for enhanced training throughput.

Related Skills

Explore More Software Engineering Skills→
cxnaivecxnaive

distributed-llm-pretraining-torchtitan

Configure and launch distributed LLM pretraining with PyTorch 4D parallelism.

Community
Advanced
ScientiaCapitalScientiaCapital
1

unsloth-finetuning

Fine-tune Llama, Mistral, Phi, or Gemma models with LoRA and 4-bit quantization.

Community
Intermediate
BermudaLocalsBermudaLocals
1

distributed-llm-pretraining-torchtitan

Enable PyTorch-native distributed LLM pretraining with 4D parallelism across 8 to 512+ GPUs.

Community
Advanced
SignmanalSignmanal
1

distributed-llm-pretraining-torchtitan

Configure and launch distributed LLM pretraining with TorchTitan 4D parallelism.

Community
Advanced