add-source

To add a new data source to PhysicsNeMo Curator, you must discover dataset properties like file format and schema, implement source logic using Python and fsspec, and register the source with the Curator pipeline for data processing integration.

When integrating datasets with PhysicsNeMo Curator, you can construct Mesh, DataArray, or AtomicData output types from raw datasets, supporting scientific and engineering domains like CAE, weather, climate, and molecular dynamics.

PhysicsNeMo Curator supports adding remote data sources from HuggingFace and S3, as well as local files. It utilizes Python libraries and fsspec backends to handle file discovery and caching across diverse data organizations.

Before integrating a local dataset into the Curator pipeline, you need familiarity with dataset properties including domain, spatial dimensions, and schema, alongside a Python environment with torch and fsspec dependencies installed.

The best way to handle file discovery for remote datasets in PhysicsNeMo Curator is using fsspec backend options and caching strategies, which efficiently manage diverse data organizations across remote storage systems like S3.

PhysicsNeMo Curator handles data source registration by guiding users through discovery questions covering domain, file format, and schema, then integrating the implemented source logic directly with the Curator pipeline for efficient data processing.