eval-worker

Writes and executes sandboxed ML training scripts and logs metrics to results.tsv.

Updated Aug 28, 2026
One-click install
npx skills add https://github.com/Its-Atharva-Gupta/forge-researcher --skill eval-worker-its-atharva-gupta
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: eval-worker
Source: https://github.com/Its-Atharva-Gupta/forge-researcher/tree/main/skills/eval_worker
Command: npx skills add https://github.com/Its-Atharva-Gupta/forge-researcher --skill eval-worker-its-atharva-gupta

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running ML experiments manually requires writing training code, executing it safely, and tracking metrics by hand. This Skill automates the full experiment loop inside an isolated sandbox so every trial is reproducible and logged. ## Core Features & Use Cases - Dynamic Script Generation: Writes training and evaluation code into workspace/experiment_{trial_id}.py based on a hypothesis specification and dataset target. - Sandboxed Execution: Runs each experiment inside TrueForge's isolated container sandbox within a 2-minute time budget. - Structured Metrics Logging: Parses val_loss, val_acc, val_f1, and val_auc and appends each trial as a row to workspace/results.tsv, marking regressed trials as DISCARDED. - Use Case: A research agent proposes a new model configuration; the Skill generates the training script, runs it in the sandbox, and appends the resulting metrics to results.tsv for downstream plotting and paper writing. ## Quick Start Run an experiment for the given hypothesis and dataset target, then append the validation metrics to workspace/results.tsv.

Frequently Asked Questions about eval-worker

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run ML experiments in a sandboxed environment?

Provide a hypothesis specification and dataset target, and the Skill writes a training script to workspace/experiment_{trial_id}.py and executes it inside TrueForge's isolated container sandbox. Each trial must complete within a 2-minute time budget.

How are experiment metrics logged to results.tsv?

After execution, the Skill parses val_loss, val_acc, val_f1, and val_auc from the run and appends them as a row to workspace/results.tsv with columns iteration, description, metrics, and status. This creates a structured, append-only experiment log.

What happens when an experiment regresses compared to previous trials?

If a trial regresses instead of improving the current best result, the Skill logs the row with status=DISCARDED and does not retain the checkpoint. Only trials that improve the state of the art keep their checkpoints.

What is the time limit for each experiment trial?

Each trial has a total time budget of 2 minutes or less, covering script execution inside the sandbox. Experiments must be designed to fit within this constraint.

Can I use this with cloud GPU execution instead of a local sandbox?

The Skill's contract specifies execution inside TrueForge's isolated container sandbox. The broader ForgeResearcher harness routes compute to Kaggle Cloud GPU or a local sandbox, but this Skill itself defines the sandboxed execution protocol.