datagen-launch

Launch datagen trace-generation jobs on HPC clusters and upload trajectories to HuggingFace.

276|37|Updated Dec 5, 2025

Search Tags:#hpc #cluster-management #harbor #daytona #trace-generation #datagen #huggingface-datasets

Ready to run

npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill datagen-launch

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: datagen-launch
Source: https://github.com/open-thoughts/OpenThoughts-Agent/tree/main/.claude/skills/datagen-launch
Command: npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill datagen-launch

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Launch a datagen (trace-generation) job on HPC clusters by coordinating a Python entrypoint for datagen so that a vLLM endpoint is started, Harbor/Daytona trace rollouts are executed, and the resulting trajectories are uploaded to a HuggingFace dataset repository.

Core Features & Use Cases

End-to-end datagen workflow: extract tasks from a parquet, submit the datagen job, monitor its progress, and consolidate generated trajectories for HF storage.
Cluster-agnostic guidance: documents shared flow and guardrails while deferring cluster-specific details to .claude/ops/<cluster>.
Interoperability with sibling skills: datagen-launch-iris, datagen-job-cleanup, and datagen-reduce-dataset-snapshots enable end-to-end lifecycle management.

Quick Start

Launch a datagen job by running the launcher with your configured datagen_config and trace harbor settings.

Frequently Asked Questions about datagen-launch

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I launch trace generation jobs on HPC clusters and upload the results to HuggingFace?▼

You can launch an end-to-end datagen trace-generation job by using a Python entrypoint to extract tasks from a parquet file, provision a vLLM endpoint, orchestrate Harbor/Daytona trace rollouts, and upload the resulting trajectories to a HuggingFace dataset repository.

What configurations are needed to run a datagen trace-generation job on an HPC cluster?▼

Running a datagen trace-generation job requires a datagen_config YAML, a trace_harbor_config YAML, a Daytona API key environment, and proper input paths. The workflow enforces guardrails to prevent misconfiguration and ensure correct environment handling.

Can I use vLLM with Daytona and Harbor for HPC cluster trace rollouts?▼

Yes, the datagen workflow provisions a vLLM endpoint and uses Daytona Harbor run orchestration to execute trace rollouts on HPC clusters. It coordinates these components to generate trajectories and consolidate them for HuggingFace storage.

How do I extract tasks from a parquet file for HPC datagen trace generation?▼

The datagen workflow performs task-parquet extraction as its first step, pulling tasks from the parquet input before submitting the datagen job to the HPC cluster. Proper input paths are required and enforced by workflow guardrails.

Does this datagen workflow support cluster-specific configurations for different HPC environments?▼

Yes, the workflow is cluster-agnostic and documents shared flow and guardrails while deferring cluster-specific details to .claude/ops/<cluster>. This allows per-cluster configuration management across different HPC environments.

What sibling skills are compatible with this datagen launch workflow for lifecycle management?▼

The datagen launch workflow interoperates with sibling skills datagen-launch-iris, datagen-job-cleanup, and datagen-reduce-dataset-snapshots to enable end-to-end datagen lifecycle management from job submission through cleanup and dataset consolidation.