datagen-launch

You can launch an end-to-end datagen trace-generation job by using a Python entrypoint to extract tasks from a parquet file, provision a vLLM endpoint, orchestrate Harbor/Daytona trace rollouts, and upload the resulting trajectories to a HuggingFace dataset repository.

Running a datagen trace-generation job requires a datagen_config YAML, a trace_harbor_config YAML, a Daytona API key environment, and proper input paths. The workflow enforces guardrails to prevent misconfiguration and ensure correct environment handling.

Yes, the datagen workflow provisions a vLLM endpoint and uses Daytona Harbor run orchestration to execute trace rollouts on HPC clusters. It coordinates these components to generate trajectories and consolidate them for HuggingFace storage.

The datagen workflow performs task-parquet extraction as its first step, pulling tasks from the parquet input before submitting the datagen job to the HPC cluster. Proper input paths are required and enforced by workflow guardrails.

Yes, the workflow is cluster-agnostic and documents shared flow and guardrails while deferring cluster-specific details to .claude/ops/<cluster>. This allows per-cluster configuration management across different HPC environments.

The datagen launch workflow interoperates with sibling skills datagen-launch-iris, datagen-job-cleanup, and datagen-reduce-dataset-snapshots to enable end-to-end datagen lifecycle management from job submission through cleanup and dataset consolidation.