skills.rest
gabrielfruet
gabrielfruetCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

pytorch-nan-debugging

Identifies root causes of non-finite values in PyTorch training runs.

3|Updated Apr 21, 2024
Search Tags:#pytorch#amp#distributed-training#deep-learning#autograd#training-stability#nan-debugging
One-click install
npx skills add https://github.com/gabrielfruet/.dotfiles --skill pytorch-nan-debugging
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: pytorch-nan-debugging
Source: https://github.com/gabrielfruet/.dotfiles/tree/main/pi/agent/skills/pytorch-nan-debugging
Command: npx skills add https://github.com/gabrielfruet/.dotfiles --skill pytorch-nan-debugging

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill eliminates the guesswork and hours of manual debugging required to track down the root cause of non-finite values, exploding losses, and numerical collapse that break PyTorch deep learning training runs.

Core Features & Use Cases

  • Structured Triage Workflow: Guides you through initial anomaly detection setup, precision checks, and stable run comparison to isolate the failure point.
  • Safe Isolated Debug Reruns: Instructs you to resume from suspect checkpoints in a fresh temporary directory to avoid corrupting original training outputs.
  • Precise Localization: Uses forward/backward hooks to identify the exact module and first non-finite tensor (input, output, or gradient) causing the failure.
  • Use Case: If your large language model training suddenly outputs NaN loss halfway through an epoch, use this Skill to systematically identify the problematic layer, get a minimal reproduction command, and determine the likely root cause.

Quick Start

Use the pytorch-nan-debugging skill to localize the first non-finite value causing your PyTorch training run to crash with NaN loss.

Frequently Asked Questions about pytorch-nan-debugging

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug NaN loss in a PyTorch deep learning training run?▼

To debug NaN loss in PyTorch, apply a structured triage workflow using autograd anomaly detection and forward/backward module hooks to isolate the exact layer and first non-finite tensor causing the numerical collapse.

Why does my PyTorch distributed training crash with NaN values halfway through an epoch?▼

PyTorch distributed training crashes with NaN values due to DDP instability or precision failures, requiring a systematic localization workflow that uses module hooks to identify the exact layer and first non-finite tensor causing the failure.

How do I find the root cause of non-finite values in PyTorch AMP training?▼

Finding the root cause of non-finite values in PyTorch AMP training involves precision checks and stable run comparisons to isolate the failure point, combined with safe isolated debug reruns to generate minimal reproduction commands.

What is the best way to track down exploding losses in PyTorch without corrupting my training outputs?▼

The best way to track down exploding losses without corrupting outputs is to resume from suspect checkpoints in a fresh temporary directory, applying isolated debug reruns and autograd anomaly detection to safely identify the root cause.

Does this NaN debugging workflow apply to checkpoint resumption errors in PyTorch?▼

Yes, this NaN debugging workflow applies to checkpoint resumption errors in PyTorch by using bisection and isolated debug reruns from suspect checkpoints to systematically identify the problematic module causing the training instability.

Can I use forward and backward hooks to identify which PyTorch module produces non-finite gradients?▼

Yes, you can use forward and backward hooks to precisely identify the exact PyTorch module and the first non-finite tensor, whether it is an input, output, or gradient, that is causing your training run to crash.

Related Skills

Explore More Software Engineering Skills→
ragnarokhaaragnarokhaa
4

pytorch-lightning

Automates PyTorch model training and streamlines distributed training.

Community
Intermediate
nishide-devnishide-dev

ml-debug

Diagnose ML training failures and generate actionable debug reports from logs.

Community
Intermediate
JNZader-VaultJNZader-Vault

pytorch

Trains PyTorch and Lightning autoencoder and Transformer models for sensor data analysis and forecasting.

Official
Advanced
SnakeOSnakeO
9

debug:pytorch

Diagnose and fix PyTorch training issues across CUDA, gradients, and data pipelines.

Community
Intermediate