skills.rest
wshobson
wshobsonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

spark-training-gotchas

Diagnose ten known failure modes for ML training on NVIDIA DGX Spark hardware.

39.3k|4.2k|Updated Jul 24, 2025
Search Tags:#Troubleshooting#CUDA#NVIDIA#DGX Spark#ml training#Preflight#gb10
One-click install
npx skills add https://github.com/wshobson/agents --skill spark-training-gotchas
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spark-training-gotchas
Source: https://github.com/wshobson/agents/tree/main/plugins/dgx-spark-ops/skills/spark-training-gotchas
Command: npx skills add https://github.com/wshobson/agents --skill spark-training-gotchas

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, and includes references (resource) and assets (resource) components.

What problem does it solve?

Training runs on NVIDIA DGX Spark (GB10, Grace Blackwell SM121, 128GB unified memory) fail in ten recurring, non-obvious ways: CUDA ABI mismatches, OOMs below the 128GB limit, thermal throttling, bandwidth ceilings, and broken parallelism strategies. This Skill provides preflight checks and diagnosis for each failure mode before or during long training jobs.

Core Features & Use Cases

  • Ten Numbered Gotchas (G1-G10): Covers CUDA 12/13 ABI mismatch, flash-attn pitfalls, UMA OOM, thermal throttling, bandwidth ceilings, memory contention, NVFP4 vs FP8 performance, stale playbooks, container drift, and dual-Spark parallelism limits.
  • Automated Preflight Script: Runs assets/preflight.sh to check G1, G3, G4, G7, and G9 with PASS/FAIL/WARN/INFO output lines keyed by G-number.
  • Runnable Diagnostic Commands: references/gotcha-checks.md provides read-only check commands per gotcha, including the Unsloth flash-attn monkeypatch override.
  • Use Case: Before launching a multi-hour fine-tuning run on DGX Spark, run the preflight script to confirm the torch build targets CUDA 13, memory headroom is real, and the environment is containerized.

Quick Start

Run the preflight checks and diagnose why my training run on DGX Spark is failing or running slower than expected.

Frequently Asked Questions about spark-training-gotchas

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
Why does training on DGX Spark OOM below the 128GB memory limit?▼

Unified memory means mmap and the CUDA allocator double-count pages during safetensors model loading, so OOM occurs while nvidia-smi still shows headroom. Read real pressure with free -g, then drop the page cache with sync and echo 3 to /proc/sys/vm/drop_caches between runs.

How do I fix undefined symbol or segfault errors on DGX Spark?▼

These errors indicate a CUDA 12/13 ABI mismatch because most PyPI wheels link libcudart.so.12 while Spark ships CUDA 13. Reinstall torch from download.pytorch.org/whl/cu130 or use a matched NGC container, and verify with torch.version.cuda reporting 13.x.

Should I use FP8 or NVFP4 for inference on DGX Spark?▼

Stay on FP8 unless your kernels are compiled for the sm_121a target. SM121 lacks the native cvt.e2m1x2 conversion path, so NVFP4 runs roughly 32 percent slower than FP8 without it. Confirm hardware capability is (12, 1) and check the kernel build's target arch.

Can I use tensor parallelism across two DGX Spark machines?▼

No, dual-Spark setups support only DDP or FSDP, never tensor parallelism. The ConnectX-7 link handles gradient and parameter sync but is too thin for TP's fine-grained traffic, causing hangs or severe slowdowns. Search your configs for tensor_parallel or tp_size settings before launching.

Why does flash-attn fail to install on DGX Spark with pip?▼

No aarch64/sm_121 wheel exists for bare pip installs, so building flash-attn yourself fails or hangs. NGC PyTorch containers ship a working SM121 flash-attn build, but watch for Unsloth silently preferring it over an explicitly requested SDPA attention implementation.

Why does DGX Spark training slow down partway through a long run?▼

Sustained power draw caps around 100W versus the 240W rated figure, causing thermal throttling or reboots on multi-hour runs. Sample temperature and power with nvidia-smi every 5 seconds for 10-15 minutes; a power plateau near 100W with climbing temperature confirms throttling.

Related Skills

Explore More Software Engineering Skills→
jayll1303jayll1303
17

docker-gpu-setup

Build GPU-enabled Docker containers with NGC base images and uv.

Community
Advanced
sheawinklersheawinkler
78

axolotl

Fine-tune large language models with YAML configurations for LoRA, DPO, and GRPO.

Community
Advanced
hhhi21ghhhi21g

unsloth

Fine-tune AI models with Unsloth for faster training and reduced memory usage.

Community
Advanced
osoleveosoleve
7

dgx-spark-vllm

Optimize vLLM inference deployment on DGX Spark clusters.

Community
Advanced