skills.rest
open-thoughts
open-thoughtsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

datagen-job-cleanup

Upload verified datagen traces to HF and free disk space.

276|37|Updated Dec 5, 2025
Search Tags:#validation#cleanup#upload#hpc#trace-generation#datagen
One-click install
npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill datagen-job-cleanup
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: datagen-job-cleanup
Source: https://github.com/open-thoughts/OpenThoughts-Agent/tree/main/.claude/skills/datagen-job-cleanup
Command: npx skills add https://github.com/open-thoughts/OpenThoughts-Agent --skill datagen-job-cleanup

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

After a datagen (trace-generation) job finishes on an HPC cluster, traces may remain on disk with incomplete or failed uploads. This Skill streamlines uploading verified traces to HF (penfever org) and reclaiming disk space, handling the common 1-level trace_jobs nesting and ensuring safe cleanup of per-run directories.

Core Features & Use Cases

  • Upload traces to HF penfever/org after a datagen job finishes (COMPLETED or TIMEOUT).
  • Validate trace integrity and completeness before upload (checks like avg_turns and exception counts) and ensure non-empty datasets.
  • Clean up local run directories and temporary task directories, with support for chunked runs consolidation and Daytona snapshot handling.

Quick Start

Run the cleanup workflow after a datagen job completes to upload traces to HF and reclaim disk space.

Frequently Asked Questions about datagen-job-cleanup

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate trace upload and disk cleanup after HPC datagen jobs finish?▼

Automate post-run trace cleanup by uploading verified traces to HF and freeing disk space for HPC datagen jobs with COMPLETED or TIMEOUT statuses.

What validation checks are applied before uploading traces to HF?▼

Trace validation checks verify avg_turns and exception rate metrics while ensuring non-empty datasets before posting traces to the penfever HF repository.

Does the cleanup workflow support nested trace_jobs directories from datagen runs?▼

The cleanup workflow supports one-level trace_jobs nesting to properly consolidate chunked runs and handle per-run directory structures during HPC trace generation.

Can I use this to safely delete Daytona snapshots after trace generation?▼

Daytona snapshots are safely cleaned up during the post-run workflow, which handles temporary task directories and reclaiming disk space after successful trace uploads.

What happens to local run directories after uploading traces to the HF repository?▼

Local run directories are cleaned up after traces are successfully uploaded and verified, which reclaims disk space by removing completed trace generation task files.

Related Skills

Explore More Data & Analytics Skills→
open-thoughtsopen-thoughts
276

datagen-launch

Launch datagen trace-generation jobs on HPC clusters and upload trajectories to HuggingFace.

Official
Advanced
huggingfacehuggingface
10.9k

hugging-face-cli

Manage Hugging Face Hub resources via the hf CLI.

Official
Intermediate
open-thoughtsopen-thoughts
276

datagen-launch-iris

Orchestrate datagen jobs on Marin's Iris TPU cluster from launch to cleanup.

Official
Advanced
cpich3gcpich3g

hugging-face-cli

Automate Hugging Face Hub workflows using the hf CLI.

Community
Intermediate