skills.rest
niznik-dev
niznik-devCOMMUNITY
Explore all skills by this author →
GitHub

Design Experiment

Plan LLM fine-tuning and evaluation experiments with YAML summaries and JSONL audit logs.

16|1|Updated Apr 4, 2025
Search Tags:#automation#planning#research#experiment design#evaluation#fine-tuning#LLM
One-click install
npx skills add https://github.com/niznik-dev/cruijff_kit --skill design-experiment
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: Design Experiment
Source: https://github.com/niznik-dev/cruijff_kit/tree/main/.claude/skills/design-experiment
Command: npx skills add https://github.com/niznik-dev/cruijff_kit --skill design-experiment

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torchtune, wandb, h5py, inspect-ai, datasets, peft, transformers, scikit-learn, matplotlib.

What problem does it solve?

Manually planning complex LLM fine-tuning and evaluation experiments is time-consuming, error-prone, and difficult to document. This Skill automates the entire planning process, ensuring consistency, verifying resources, and providing clear documentation, so you can focus on the science.

Core Features & Use Cases

  • Automated Experiment Planning: Guides you through defining variables, resources, and evaluation strategies for LLM fine-tuning.
  • Resource Verification: Automatically checks for the existence of models, datasets, and evaluation scripts on your system.
  • Compute Estimation: Provides accurate estimates for training time, evaluation time, and disk space, often based on prior runs.
  • Self-Documenting Plan: Generates a comprehensive experiment_summary.md with all configurations and status tracking, ready for review.
  • Use Case: Before starting a new research project comparing different LoRA ranks for a specific LLM, use this skill to generate a detailed plan, verify all necessary files are in place, and get accurate time/cost estimates, all documented automatically.

Quick Start

To start designing your experiment (this skill is interactive):

claude design-experiment

Follow the prompts to define your experiment's parameters.

A detailed experiment_summary.md will be created upon completion,

outlining your entire research plan.

Frequently Asked Questions about Design Experiment

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I plan LLM fine-tuning experiments with automated resource verification?▼

Design Experiment automates the entire planning process for LLM fine-tuning by guiding you through parameter definition, verifying that models and datasets exist on your system, estimating compute requirements, and generating a comprehensive experiment summary—eliminating manual planning errors and documentation gaps.

Can I use this skill to estimate training time and disk space for fine-tuning?▼

Yes. Design Experiment provides compute and storage estimates based on your experiment configuration, often informed by prior runs, so you can accurately forecast training duration and disk requirements before starting.

What output does the experiment planning process generate?▼

Design Experiment produces experiment_summary.yaml for structured configuration, experiment_summary.md for human review, and design-experiment.jsonl as an audit log—all automatically documented and ready for reproducibility.

Does this work with torchtune and inspect-ai for fine-tuning and evaluation?▼

Yes. Design Experiment integrates with torchtune for training workflows and inspect-ai for evaluation, enforcing a three-stage pattern (parameter selection, validation, generation) and creating integration points with scaffold-experiment and run-experiment skills.

Do I need research experience to design experiments with this skill?▼

No. The interactive prompts guide you through defining variables, resources, and evaluation strategies step by step, automating consistency checks and resource verification so you can focus on your research goals rather than manual configuration.

What happens if required models or datasets are missing from my system?▼

Design Experiment automatically detects missing models, datasets, and evaluation scripts during resource verification, alerting you before planning completes so you can resolve dependencies before execution.

Related Skills

Explore More Education & Research Skills→
neuron-oneneuron-one
25

ml-training

Automate end-to-end HuggingFace model fine-tuning workflows with sft, dpo, or grpo.

Community
Advanced
niznik-devniznik-dev
16

Run Experiment

Orchestrate SLURM fine-tuning jobs with torchtune and evaluate with inspect-ai.

Community
Advanced
JeffallanJeffallan
10.9k

fine-tuning-expert

Automate fine-tuning workflows for large language models using LoRA/QLoRA and PEFT methods.

Community
Advanced
reggiechan74reggiechan74
5

agent-skills

Create modular agent skills with progressive disclosure and tool-access controls.

Community
Intermediate