nvalchemi-data-storage

To efficiently persist and batch atomic data for GPU training, use a Zarr-backed storage pipeline that writes AtomicData to a Zarr store and reads it back through a Dataset and DataLoader for batched iteration.

The best way to load atomic data from a Zarr store into a DataLoader is through a modular pipeline. A Reader component fetches the persisted data, passes it to a Dataset, and the DataLoader handles device transfer, prefetching, and batched iteration for model evaluation and training.

Yes, you can use this Zarr data storage pipeline for GPU-accelerated evaluation workflows. The pipeline supports device transfer and prefetching, enabling efficient loading and batching of AtomicData for both model training and evaluation.

You write AtomicData objects to a Zarr dataset by using a Writer component within the storage pipeline. This persists the atomic data efficiently to a Zarr store, allowing a Reader to fetch the data later for analysis or model training.

Yes, this atomic data storage approach supports prefetching and device transfer. The DataLoader component within the Zarr-backed pipeline is designed to handle GPU-friendly data access, enabling efficient batched iteration during model training and evaluation.