finetune-hyperparameter-sweep

Sweeps learning rates across fine-tuning runs until task pass criteria are met.

75|7|Updated May 2, 2026
One-click install
npx skills add https://github.com/zjunlp/Mechanist --skill finetune-hyperparameter-sweep-zjunlp
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: finetune-hyperparameter-sweep
Source: https://github.com/zjunlp/Mechanist/tree/main/skills/experiment-tips/finetune-hyperparameter-sweep
Command: npx skills add https://github.com/zjunlp/Mechanist --skill finetune-hyperparameter-sweep-zjunlp

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Fine-tuning runs often adopt a learning rate copied from a paper or hard-coded in a plan, then get judged by healthy-looking loss or reward curves that prove nothing. This Skill enforces a discipline where the learning rate is always swept over a wide range and judged only against explicit pass criteria. ## Core Features & Use Cases - Mandatory LR Sweeping: Treats any fixed or paper-copied learning rate as a single grid point and requires sweeping a wide range for full fine-tuning, LoRA, QLoRA, DoRA, and other PEFT adapters. - Criteria-Based Acceptance: Accepts a learning rate only when the run meets the pass criteria in task.md or the milestone's claim criteria, rejecting loss, grad norm, and reward curves as evidence. - Iterative Re-Run Protocol: When criteria are not met, changes the learning rate and re-runs until an LR passes, covering SFT, DPO, GRPO, PPO, and RLHF objectives. - Use Case: While running an SFTTrainer job that converges smoothly but fails the task's evaluation criteria, apply this Skill to reject the run, sweep the learning rate across a wide range, and re-run until a configuration passes. ## Quick Start Sweep the learning rate over a wide range for this LoRA fine-tune and re-run until the task.md pass criteria are met.

Frequently Asked Questions about finetune-hyperparameter-sweep

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose a learning rate for LLM fine-tuning?

Treat any proposed learning rate, including one copied from a reference paper, as a single grid point and sweep a wide range instead. Accept a value only when the run meets the pass criteria defined in task.md or the milestone's claim criteria.

Does this LR sweep protocol work with LoRA and QLoRA adapters?

Yes, the protocol covers full fine-tuning and PEFT adapters including LoRA, QLoRA, and DoRA. It applies across SFT, DPO, and GRPO/PPO/RLHF objectives, triggered by configurations like LoraConfig or peft_config.

Can I trust a converged loss curve as evidence fine-tuning worked?

No. A converged loss, healthy grad norm, or rising reward curve is never acceptance evidence under this protocol. The only valid test is whether the run meets the explicit pass criteria in task.md or the milestone's claim criteria.

What should I do when fine-tuning fails the pass criteria?

Change the learning rate and re-run, repeating until an LR passes the criteria. Only after the entire LR range has failed should you modify anything else in the training setup.

When does this fine-tuning skill get triggered?

It fires on any fine-tuning task, especially when a plan already fixes a learning rate or copies one from a paper. Triggers include learning_rate, SFTTrainer, DPOTrainer, GRPOTrainer, LoraConfig, peft_config, SFT, DPO, GRPO, RLHF, QLoRA, DoRA, and PEFT.