skills.rest
dongzhuoyao
dongzhuoyaoCOMMUNITY
Explore all skills by this author →
Website𝕏@vtaohuGitHub

gpu-training-acceleration

Optimize PyTorch training speed and memory usage on CUDA GPUs.

10|1|Updated Feb 21, 2026
Search Tags:#performance#gpu#pytorch#triton#mixed-precision#gradient-checkpointing#torch.compile
One-click install
npx skills add https://github.com/dongzhuoyao/tao-research-skills --skill gpu-training-acceleration
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: gpu-training-acceleration
Source: https://github.com/dongzhuoyao/tao-research-skills/tree/main/gpu-training-acceleration
Command: npx skills add https://github.com/dongzhuoyao/tao-research-skills --skill gpu-training-acceleration

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Optimize PyTorch training speed and reduce GPU memory usage on CUDA GPUs by providing actionable, config-gated acceleration patterns, mixed precision guidance, fused optimizer kernels, Triton kernel fusion, and monitoring practices. It helps diagnose low GPU utilization, avoid out-of-memory failures, and safely adopt new compiler or kernel features without destabilizing training runs.

Core Features & Use Cases

  • Config-gated acceleration: Patterns to gate features like torch.compile, fused optimizers, and gradient checkpointing behind configuration so they can be toggled and logged.
  • Precision and kernel guidance: Advice for TF32, bf16/fp16 mixed precision, cuDNN autotuning, and when to prefer fused kernels or Triton implementations.
  • Failure handling and telemetry: Safe defaults, try/except fallbacks for torch.compile, and recommendations to always log acceleration state for reproducibility and debugging.
  • Memory and dataflow patterns: Strategies for latent-space training, contiguous memory enforcement, empty_cache placement, and NVCC build flags for custom CUDA extensions.
  • Use Case: Speed up transformer or generative model training on multi-GPU clusters by enabling TF32, using bf16 mixed precision where safe, compiling stable decoder submodules, and switching to fused AdamW for optimizer speedups.

Quick Start

Run a short fastrun with TF32 enabled, mixed precision set to bf16, fused optimizer turned on, and compile only stable-shape decoder submodules to validate throughput and memory before full-scale training.

Frequently Asked Questions about gpu-training-acceleration

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I optimize PyTorch training speed and reduce GPU memory usage on CUDA?▼

To optimize PyTorch training speed and reduce GPU memory, you can enable TF32, use bf16/fp16 mixed precision autocast, apply gradient checkpointing, and switch to fused optimizers like AdamW to maximize throughput and memory efficiency.

What is the best way to configure torch.compile for submodules without destabilizing training?▼

The best way to configure torch.compile safely is to gate it behind configuration with try/except fallbacks and compile only stable-shape decoder submodules, ensuring you log the acceleration state for reproducibility and debugging.

Why does my PyTorch generative model training keep running out of GPU memory on multi-GPU clusters?▼

PyTorch generative model training runs out of GPU memory due to unoptimized dataflow. You can resolve this by enforcing contiguous memory, managing empty_cache placement, using gradient checkpointing, and applying latent-space training strategies.

Does this PyTorch GPU acceleration guidance work with HPC and Slurm clusters?▼

Yes, this PyTorch GPU acceleration guidance applies to single- and multi-GPU setups, including HPC and Slurm clusters, providing actionable patterns for mixed-precision workflows, Triton kernel fusion, and cuDNN autotuning.

When should I use Triton kernel fusion instead of standard PyTorch cuDNN operations?▼

You should use Triton kernel fusion when seeking maximum throughput beyond standard cuDNN autotuning, applying custom fusion patterns for transformer and generative model training where safe, while relying on fused kernels for optimizer speedups.

Related Skills

Explore More Software Engineering Skills→
xchang1121xchang1121
6

triton-cuda-examples-torch

Implement Triton CUDA kernels for PyTorch neural network layers.

Community
Advanced
pacphipacphi
14

PyTorch ML

Define and train PyTorch neural networks with CUDA acceleration.

Community
Advanced
LancerLabLancerLab

croq-dsl-cute-cpp

Build, run, and profile CuTe/CUTLASS C++ GPU kernels with nvcc.

Official
Advanced
longkeyylongkeyy
1

content-discovery

Automate content discovery across ArXiv, GitHub, and HuggingFace with semantic filtering and publishing.

Community
Advanced