skills.rest
niznik-dev
niznik-devCOMMUNITY
Explore all skills by this author →
GitHub

Scaffold Experiment

Create directories, YAML configs, and SLURM scripts for LLM fine-tuning runs.

16|1|Updated Apr 4, 2025
Search Tags:#automation#research#configuration#experiment setup#fine-tuning#SLURM#LLM
One-click install
npx skills add https://github.com/niznik-dev/cruijff_kit --skill scaffold-experiment
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Scaffold Experiment
Source: https://github.com/niznik-dev/cruijff_kit/tree/main/.claude/skills/scaffold-experiment
Command: npx skills add https://github.com/niznik-dev/cruijff_kit --skill scaffold-experiment

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torchtune, wandb, h5py, inspect-ai, datasets, peft, transformers, scikit-learn, matplotlib.

What problem does it solve?

Manually creating directories, configuring YAML files, and generating SLURM scripts for multiple LLM fine-tuning runs is repetitive and prone to human error. This Skill automates the entire setup process, ensuring consistency and saving significant preparation time, so your experiments are ready to launch.

Core Features & Use Cases

  • Automated Directory Creation: Sets up a structured directory for each experiment run based on your experiment_summary.md.
  • Config File Generation: Creates setup_finetune.yaml for each run, pre-populated with specific parameters.
  • SLURM Script Generation: Automatically generates finetune.yaml and finetune.slurm files, making each run ready for job submission.
  • Use Case: After designing an experiment with 10 different fine-tuning runs, use this skill to automatically generate all 10 run directories, their specific configurations, and SLURM submission scripts in minutes, rather than hours of manual copying and editing.

Quick Start

Assuming an experiment_summary.md exists in the current directory:

claude scaffold-experiment

The skill will create subdirectories for each run,

populate them with config files, and generate SLURM scripts,

preparing your experiment for submission.

Frequently Asked Questions about Scaffold Experiment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate SLURM script generation for multiple LLM fine-tuning runs?▼

Scaffold Experiment automates SLURM script generation by reading your experiment_summary.yaml, creating run directories, and producing finetune.yaml and finetune.slurm files for each configuration. This eliminates manual copying and editing across multiple runs.

Can I use this to set up fine-tuning experiments with torchtune and wandb?▼

Yes. Scaffold Experiment orchestrates setup for torchtune-based fine-tuning runs and integrates wandb for experiment tracking. It generates setup_finetune.yaml configs populated with your experiment parameters, preparing runs for immediate submission.

What files do I need before running the experiment scaffold?▼

You need experiment_summary.yaml defining your runs and claude.local.md for local configuration. Scaffold Experiment validates these prerequisites, then generates all required directories and config files automatically.

How does this handle evaluation alongside fine-tuning setup?▼

Scaffold Experiment orchestrates both preparation via scaffold-torchtune and evaluation via scaffold-inspect, launching them in parallel. It records an orchestration log and final summary, coordinating the full experimental infrastructure in one pass.

Can Scaffold Experiment run experiments at scale with parallel execution?▼

Yes. It maps tools to subagents and launches preparation and evaluation tasks in parallel using the Task tool, saving significant time when setting up multiple runs simultaneously.

Related Skills

Explore More Education & Research Skills→
niznik-devniznik-dev
16

Design Experiment

Plan LLM fine-tuning and evaluation experiments with YAML summaries and JSONL audit logs.

Community
Advanced
dongzhuoyaodongzhuoyao
10

slurm-gpu-training

Submit GPU training jobs to Slurm clusters with preflight validation and monitoring.

Community
Advanced
forgefabrikforgefabrik

finetuning

Fine-tune Azure AI Foundry models with SFT, DPO, or RFT jobs.

Community
Advanced
PlagueHOPlagueHO
1

skill-creator

Scaffold Agent Skill directories with YAML frontmatter and optional assets.

Community
Intermediate