skills.rest
xu1713
xu1713COMMUNITY
Explore all skills by this author →
GitHub

huggingface-accelerate

Adds distributed training support to PyTorch scripts with four lines of code.

Updated Jun 5, 2026
Search Tags:#pytorch#distributed training#huggingface#fsdp#mixed precision#deepspeed#multi-gpu
One-click install
npx skills add https://github.com/xu1713/openhorse --skill huggingface-accelerate-xu1713
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: huggingface-accelerate
Source: https://github.com/xu1713/openhorse/tree/main/openhorse/openhorse/optional-skills/mlops/accelerate
Command: npx skills add https://github.com/xu1713/openhorse --skill huggingface-accelerate-xu1713

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires accelerate, torch, transformers, and includes references (resource) components.

What problem does it solve? Scaling PyTorch training from a single GPU to multi-GPU or multi-node clusters normally requires rewriting code for DDP, DeepSpeed, FSDP, or Megatron, plus manual device placement and mixed precision handling. This Skill provides a unified API that handles all of that automatically. ## Core Features & Use Cases - Minimal Code Changes: Convert any PyTorch training loop to distributed training by adding just 4 lines using the Accelerator class. - Unified Backend Support: Run the same script on DDP, DeepSpeed ZeRO, FSDP, or Megatron-LM with interactive configuration via accelerate config. - Mixed Precision & Memory Optimization: Enable FP16, BF16, or FP8 training, gradient accumulation, and CPU/NVMe offloading with simple flags. - Use Case: You have a single-GPU training script for a transformer model and need to scale it across 8 GPUs with BF16 mixed precision. Add the Accelerator, call prepare() on your model, optimizer, and dataloader, then launch with accelerate launch --multi_gpu --num_processes 8 train.py. ## Quick Start Convert my PyTorch training script to run on multiple GPUs using HuggingFace Accelerate with BF16 mixed precision.

Frequently Asked Questions about huggingface-accelerate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert a PyTorch script to multi-GPU training?▼

Add four lines with HuggingFace Accelerate: import Accelerator, instantiate it, call accelerator.prepare() on your model, optimizer, and dataloader, and replace loss.backward() with accelerator.backward(loss). Then launch with accelerate launch --multi_gpu train.py.

What is the difference between DeepSpeed, FSDP, and DDP in Accelerate?▼

DDP replicates the model on each GPU and suits smaller models. DeepSpeed ZeRO and FSDP shard optimizer states, gradients, and parameters across GPUs for large models. Accelerate provides one API so the same code runs on any backend selected via accelerate config.

Does Accelerate support mixed precision training with BF16 or FP8?▼

Yes, pass mixed_precision='bf16', 'fp16', or 'fp8' to the Accelerator constructor. BF16 needs no gradient scaling and is more stable, FP16 uses GradScaler for older GPUs, and FP8 targets H100 hardware with TransformerEngine.

Why is gradient accumulation not working with Accelerate?▼

Gradient accumulation requires wrapping the training step in the accelerator.accumulate(model) context manager. Set gradient_accumulation_steps in the Accelerator constructor, and the context manager handles synchronization and optimizer stepping automatically.

Can Accelerate train large models with Megatron tensor parallelism?▼

Yes, Accelerate integrates Megatron-LM through MegatronLMPlugin, supporting tensor parallelism, pipeline parallelism, and sequence parallelism. Configure degrees via accelerate config and launch with the standard accelerate launch command.

When should I use PyTorch Lightning or Ray Train instead of Accelerate?▼

Use PyTorch Lightning when you need built-in callbacks and high-level training abstractions, or Ray Train for multi-node orchestration with hyperparameter tuning. Accelerate fits when you want minimal code changes and direct control over the training loop.

Related Skills

Explore More Software Engineering Skills→
OpenLAIROpenLAIR
1.0k

training-llms-megatron

Train large language models with Megatron-Core parallelism on multi-node GPU clusters.

Official
Advanced
VMAFxVMAFx
3

bisect-model-quality

Locate the first ONNX checkpoint that drops below a PLCC, SROCC, or RMSE quality gate.

Official
Advanced
Ven0m0Ven0m0
6

cpu-gpu-performance

Monitor CPU and GPU utilization and throttle resource usage during intensive operations.

Community
Advanced
tadod12tadod12

ray-train

Orchestrate distributed ML training across multi-node clusters with Ray Train.

Community
Advanced