distributed-llm-pretraining-torchtitan

Plan and configure distributed LLM pretraining for PyTorch workloads.

Updated May 14, 2026

Search Tags:#checkpointing #distributed training #pipeline parallelism #torchtitan #float8 #fsdp2 #llm pretraining

Ready to run

npx skills add https://github.com/SethyPagna/Secretary-Jarvis --skill distributed-llm-pretraining-torchtitan-sethypagna

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: distributed-llm-pretraining-torchtitan
Source: https://github.com/SethyPagna/Secretary-Jarvis/tree/main/src/capabilities/optional-skills/mlops/torchtitan
Command: npx skills add https://github.com/SethyPagna/Secretary-Jarvis --skill distributed-llm-pretraining-torchtitan-sethypagna

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It removes the complexity of launching and maintaining large-scale PyTorch LLM pretraining jobs by giving you a single workflow for model setup, parallelism, checkpointing, and performance tuning.

Core Features & Use Cases

4D parallelism: Coordinate FSDP2, tensor parallelism, pipeline parallelism, and context parallelism for 8 to 512+ GPU training runs.
Performance optimization: Enable torch.compile and Float8 training to improve throughput on H100 and newer hardware.
Checkpoint resilience: Save, resume, convert, and shard checkpoints with PyTorch Distributed Checkpoint and HuggingFace interoperability.
Custom model support: Add new model families by defining model args, parallelization logic, and optional state-dict adapters.
Use case: Train Llama 3.1, DeepSeek V3, GPT-OSS, Qwen 3, Flux, or a custom transformer across single-node or multi-node clusters.

Quick Start

Ask for a TorchTitan training plan for your target model, GPU count, and checkpointing needs, and receive a launch-ready configuration with the right parallelism and optimization settings.

Frequently Asked Questions about distributed-llm-pretraining-torchtitan

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I configure distributed LLM pretraining for PyTorch across multiple GPUs?▼

To configure distributed LLM pretraining, you coordinate 4D parallelism: FSDP2, tensor parallelism, pipeline parallelism, and context parallelism. This setup allows you to scale PyTorch training workloads efficiently across 8 to 512+ GPUs in single-node or multi-node clusters.

What is the best way to improve PyTorch LLM pretraining throughput on H100 GPUs?▼

The best way to improve pretraining throughput on H100 hardware is to enable torch.compile and Float8 training. These performance optimizations significantly increase processing speed during large-scale LLM pretraining jobs on newer hardware.

Can I resume LLM pretraining from a checkpoint and convert it for HuggingFace?▼

Yes, you can resume LLM pretraining from a checkpoint and convert it for HuggingFace. The workflow uses PyTorch Distributed Checkpoint to save, resume, convert, and shard checkpoints, ensuring full HuggingFace interoperability for your models.

Does this distributed training setup support custom transformer models?▼

Yes, this distributed training setup supports custom transformer models. You can add new model families by defining model arguments, parallelization logic, and optional state-dict adapters alongside pre-configured models like Llama 3.1 and DeepSeek V3.

How do I set up 4D parallelism for Llama 3.1 pretraining?▼

You set up 4D parallelism for Llama 3.1 pretraining by requesting a launch-ready training plan. This configuration coordinates FSDP2, tensor, pipeline, and context parallelism tailored to your specific GPU count and checkpointing requirements.