What problem does it solve?
NVIDIA DGX Spark's GB10 chip shares one 128GB unified memory pool between CPU and GPU, so nvidia-smi underreports memory pressure and jobs OOM unexpectedly, while sustained power limits cause mid-run slowdowns that look like configuration bugs. This Skill provides the planning math, OOM remediation order, and thermal monitoring workflow to run multi-hour training jobs reliably on that hardware.
Core Features & Use Cases
- UMA Memory Planning: Budget against
free -g instead of nvidia-smi, estimate weights, optimizer states, gradients, and activations with the uma-accounting worksheet, and sanity-check against known anchors like 70B QLoRA at ~40GB.
- Ordered OOM Remediation: Work the OOM Ladder in sequence — flush page cache first, then reduce batch size or packing length, then downgrade method from QLoRA to bf16 LoRA — instead of guessing.
- Thermal and Power Monitoring: Sample GPU temperature and power every 30-60 seconds with the thermal-sample.sh script and correlate CSV output with training logs to distinguish throttling from config bugs.
- Use Case: Before launching a 70B QLoRA fine-tune, estimate the footprint against the 128GB pool, start the thermal sampler, and when the run OOMs mid-load, flush the buffer cache before touching batch size.
Quick Start
Ask the AI to plan memory headroom for a training run on DGX Spark and set up thermal monitoring before launch.
