cupynumeric-parallel-data-load

To load sharded npy files with non-uniform row counts into a single cupynumeric array, this skill parses per-shard headers, allocates a destination array, partitions by tiling, and launches parallel leaf tasks across CPUs and GPUs to read the data.

The best way to perform parallel data loading for distributed cupynumeric arrays is using manual partitioning across processors and launching multi-variant leaf tasks, which reads multi-file datasets in parallel across CPU, OMP, and GPU targets.

Yes, this parallel loading approach works with Parquet, Arrow, HDF5, raw binary, and npy formats by streaming per-file reads directly inside leaf tasks to ingest sharded on-disk data efficiently.

Yes, you can use legate to ingest multi-file simulation datasets across multiple GPUs, as the skill performs manual partitioning across processors and enforces final synchronization to unify sharded data into a distributed ndarray.

Loading multi-file datasets into cupynumeric requires manual partitioning and leaf tasks because there is no built-in loader for whole datasets with varying shard sizes, necessitating manual tiling and parallel reads to populate the destination array.

A limitation of using cupynumeric for sharded data loading is that it requires manual partitioning and allocation by tiling, as no built-in loader exists for datasets with non-uniform shard row counts, demanding explicit synchronization.