sue-dataset-preparation

What problem does it solve?

Preparing datasets for large-scale ML/HPC experiments on HPC clusters often leads to inode quota exhaustion on parallel filesystems, misplaced datasets in invalid project directories, or sandbox accessibility failures that derail expensive experiment runs. This Skill eliminates those risks by enforcing strict storage, formatting, and validation rules tailored to each supported HPC backend.

Core Features & Use Cases

• Inode-safe dataset formatting: Automatically recommends and implements LMDB, WebDataset tar shards, or SquashFS layouts to avoid metadata pressure on Lustre/GPFS parallel filesystems, instead of leaving datasets as millions of small individual files.
• Backend-specific storage compliance: Enforces correct path rules for all supported HPC sandboxes (LUMI, Snellius, NM5, Brev, RunPod, AutoDL) to ensure datasets are stored in accessible, quota-valid locations rather than invalid project source trees or user home directories.
• Preflight validation and budget checks: Runs space, inode, and accessibility probes before dataset preparation to catch quota shortfalls or path issues early, preventing wasted compute on failed experiment runs.
• Use Case: A researcher preparing a 2TB ImageNet-scale dataset for a LUMI cluster can use this Skill to convert raw archives to LMDB to avoid inode exhaustion, verify the LUMI scratch quota has sufficient headroom, and write the final dataset path to the experiment runtime config for downstream training jobs.

Quick Start

Use the sue-dataset-preparation skill to convert your raw dataset archives to an inode-safe format, verify your target HPC sandbox's storage quotas, and record the final dataset path in your experiment runtime configuration.