huggingface-accelerate

Name: huggingface-accelerate
Availability: InStock
Author: Khyro

Scale PyTorch training with a unified distributed API for multi-GPU and mixed-precision workflows.

Updated Apr 2, 2026

Search Tags:#pytorch #huggingface #distributed-training #mixed-precision #fsdp #deepspeed #megatron

Ready to run

npx skills add https://github.com/JKhyro/HERMES-AGENT --skill huggingface-accelerate-jkhyro

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: huggingface-accelerate
Source: https://github.com/JKhyro/HERMES-AGENT/tree/main/optional-skills/mlops/accelerate
Command: npx skills add https://github.com/JKhyro/HERMES-AGENT --skill huggingface-accelerate-jkhyro

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill removes the complexity of distributed training setup so you can scale a single PyTorch script across one GPU, many GPUs, multiple nodes, or different mixed-precision backends without rewriting your workflow.

Core Features & Use Cases

Unified distributed API: Prepare models, optimizers, and dataloaders once, then run the same code on DDP, DeepSpeed, FSDP, or Megatron-LM.
Performance and memory controls: Use automatic device placement, mixed precision, gradient accumulation, checkpointing, and sharded optimization to train larger models more efficiently.
Production training scenarios: Ideal for upgrading a prototype to multi-GPU training, launching a multi-node fine-tune, or tuning throughput and memory for large language model workloads.

Quick Start

Use this Skill to convert a standard PyTorch training script into an Accelerate-based distributed training workflow and launch it with the right configuration for your hardware.

Frequently Asked Questions about huggingface-accelerate

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I scale PyTorch training across multiple GPUs without rewriting my script?▼

You can scale PyTorch training across multiple GPUs by using a unified distributed API that prepares your model, optimizer, and dataloader once. This approach allows the same script to run on single or multi-GPU setups without modifying your workflow.

What is the best way to run mixed-precision training with FSDP and DeepSpeed?▼

The best way to run mixed-precision training with FSDP and DeepSpeed is to use a unified API that supports both backends. You configure the launcher and sharded optimization settings once, and the API handles the backend-specific execution automatically.

Can I use a single PyTorch script for both single GPU and multi-node distributed training?▼

Yes, you can use a single PyTorch script for both single GPU and multi-node distributed training. By relying on automatic device placement and a unified API, the script scales from a prototype to a multi-node fine-tune without code changes.

Does Hugging Face Accelerate support Megatron-LM for large language model workloads?▼

Yes, Hugging Face Accelerate supports Megatron-LM for large language model workloads. It integrates Megatron-LM alongside DDP, DeepSpeed, and FSDP within its unified distributed API to manage large-scale training scenarios efficiently.

How do I configure gradient accumulation and checkpointing for distributed training?▼

You configure gradient accumulation and checkpointing for distributed training by enabling these features within the unified API's performance controls. This manages memory constraints and optimizes throughput for large models during execution.