Read and write distributed cuPyNumeric arrays as HDF5 files with Legate parallel I/O.
npx skills add https://github.com/NVIDIA/skills --skill cupynumeric-hdf5
Please help me install this Agent Skill. Skill: cupynumeric-hdf5 Source: https://github.com/NVIDIA/skills/tree/main/skills/cupynumeric-hdf5 Command: npx skills add https://github.com/NVIDIA/skills --skill cupynumeric-hdf5
Moving large distributed cuPyNumeric arrays to and from disk is error-prone: naive single-process writes bottleneck I/O, asynchronous Legate writes appear truncated to external readers, and GPU reads over ~128 MB abort on the default staging buffer. This Skill teaches agents the correct legate.io.hdf5 workflow so arrays round-trip to single .h5/.hdf5 files safely across ranks.
to_file and from_file so every rank writes its own tile into one virtual dataset, and bridge results back with cn.asarray(...).from_file_batched to load very large datasets chunk by chunk with correct offset placement of clipped boundary chunks.LEGATE_IO_USE_VFD_GDS=1 for GPU reads to avoid the 128 MB ZCMEM staging abort, with cuFile compatibility-mode fallback.to_file call, the mandatory execution fence before external readers, and the h5py prerequisite install.Ask your agent to save a cuPyNumeric array to an .h5 file with Legate's HDF5 API and load it back for verification.
High-intent search queries and answers about installing and using this skill.
Call legate.io.hdf5.to_file(array=a, path='out.h5', dataset_name='/data'), passing the cuPyNumeric ndarray directly without converting to NumPy. Follow the write with get_legate_runtime().issue_execution_fence(block=True) before any external tool opens the file.
Use from_file_batched(path, dataset_name, chunk_size), which yields one LogicalArray per chunk plus its offsets in the global shape. Place each chunk into a preallocated cn.empty array by its actual shape, since boundary chunks are clipped.
Legate I/O is asynchronous, so to_file only queues the write and h5py may open the file before it lands. Insert get_legate_runtime().issue_execution_fence(block=True) between the write and any external reader.
Yes, set LEGATE_IO_USE_VFD_GDS=1 for any run reading HDF5 into GPU memory, even without GPUDirect-capable storage since cuFile falls back to compatibility mode. The default POSIX path aborts on GPU arrays larger than the 128 MB ZCMEM staging buffer.
The legate.io.hdf5 module imports h5py at load time, and h5py is not included in the default cuPyNumeric environment. Install it with conda install -c conda-forge h5py before importing.
Avoid it for Parquet/cuDF or raw-binary layouts (use the cupynumeric-parallel-data-load skill), Zarr or S3 object-store output, .npz archives (use np.load plus cn.asarray), and plain h5py scripts with no cuPyNumeric arrays.
sayalinvidiaLoad sharded on-disk datasets into a distributed cupynumeric ndarray.
SciMate-AIAutomate ParaView HPC post-processing pipelines with pvpython and pvbatch scripts.
shushuznStore and retrieve chunked, compressed N-dimensional arrays with NumPy-like indexing.
vertivolatamAutomate Flutter backend infrastructure deployment on AWS with EKS, RDS, and Lambda.