add-filter

To implement a new data processing filter in the PhysicsNeMo Curator framework, follow a guided step-by-step process covering design considerations, pattern selection, testing, and deployment to ensure full workflow compatibility.

Available filter implementation patterns include pass-through, in-place modification, stateful with flush, and stateful with merge. These patterns guide how data is processed and integrated within existing workflows.

You test and register a custom filter for PhysicsNeMo Curator by following specific framework guidelines that verify the filter's compatibility and ensure it integrates correctly with existing data processing workflows.

Yes, PhysicsNeMo Curator supports stateful data processing filters through specific implementation patterns, specifically stateful with flush and stateful with merge, allowing for complex data transformations.

You should use the in-place modification filter pattern in PhysicsNeMo Curator when your data processing workflow requires direct alteration of the dataset without needing to pass the data through a separate output structure.