datagen-standard-launch

Generate synthetic datasets via Curator sharding or declarative data scripts.

276|37|Updated Dec 5, 2025

Search Tags:#generator #data-generation #multi-node #vllm #curator #synthetic-data #datagen

Ready to run

npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill datagen-standard-launch

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: datagen-standard-launch
Source: https://github.com/open-thoughts/OpenThoughts-Agent/tree/main/.claude/skills/datagen-standard-launch
Command: npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill datagen-standard-launch

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

datagen-standard-launch enables non-agentic data generation by producing model completions and synthetic data directly, without Harbor agent loops or Daytona sandboxes.

Core Features & Use Cases

Path 1: Curator sharded datagen (multi-node DP) using run_curator_datagen_sharded.sbatch to generate disjoint data slices across nodes with auto-resume and per-shard checkpoints.
Path 2: declarative / class-based generator scripts under data/ with generate.py and generate_abstract.py for local or HPC runs, enabling flexible data generation pipelines.
Use Case: bulk synthetic data creation for model training, evaluation, and benchmarking.

Quick Start

Run the standard datagen workflow to generate synthetic data from a model using either the multi-node Curator path or the local data/generation scripts.

Frequently Asked Questions about datagen-standard-launch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate synthetic data at scale without using agent loops?▼

Synthetic data generation without agent loops is automated by producing model completions directly through declarative scripts or Curator-based sharding. This approach enables bulk dataset creation for training and evaluation without Harbor agent overhead.

What is the best way to generate disjoint synthetic data slices across multiple nodes?▼

Generating disjoint synthetic data slices across multiple nodes is handled by Curator sharded datagen using run_curator_datagen_sharded.sbatch. This multi-node data parallel path creates separate data slices with auto-resume and per-shard checkpointing capabilities.

Can I use declarative generator scripts for local synthetic data creation?▼

Declarative generator scripts under data/ with generate.py and generate_abstract.py support local synthetic data creation. These class-based generator scripts enable flexible data generation pipelines for both local and HPC runs.

Does Curator datagen support auto-resume and per-shard checkpoints for sharded runs?▼

Curator datagen supports auto-resume and per-shard checkpoints for sharded runs. This ensures disjoint data slices are generated reliably across nodes, preventing data loss during bulk synthetic data creation.

Why use non-agentic data generation instead of Harbor agent loops for bulk dataset creation?▼

Non-agentic data generation produces model completions and synthetic datasets directly, bypassing Harbor agent loops or Daytona sandboxes. This approach is suited for bulk synthetic data creation where agent orchestration overhead is unnecessary.

When do I need multi-node Curator sharded datagen versus local generation scripts?▼

Multi-node Curator sharded datagen is needed for large-scale disjoint data slicing across distributed nodes, while local declarative generation scripts suit smaller flexible pipeline runs. Choose based on whether your synthetic data creation requires distributed data parallelism.