datagen-standard-launch

Synthetic data generation without agent loops is automated by producing model completions directly through declarative scripts or Curator-based sharding. This approach enables bulk dataset creation for training and evaluation without Harbor agent overhead.

Generating disjoint synthetic data slices across multiple nodes is handled by Curator sharded datagen using run_curator_datagen_sharded.sbatch. This multi-node data parallel path creates separate data slices with auto-resume and per-shard checkpointing capabilities.

Declarative generator scripts under data/ with generate.py and generate_abstract.py support local synthetic data creation. These class-based generator scripts enable flexible data generation pipelines for both local and HPC runs.

Curator datagen supports auto-resume and per-shard checkpoints for sharded runs. This ensures disjoint data slices are generated reliably across nodes, preventing data loss during bulk synthetic data creation.

Non-agentic data generation produces model completions and synthetic datasets directly, bypassing Harbor agent loops or Daytona sandboxes. This approach is suited for bulk synthetic data creation where agent orchestration overhead is unnecessary.

Multi-node Curator sharded datagen is needed for large-scale disjoint data slicing across distributed nodes, while local declarative generation scripts suit smaller flexible pipeline runs. Choose based on whether your synthetic data creation requires distributed data parallelism.