Configures supervised fine-tuning recipes for vision-language models on image-text datasets.
npx skills add https://github.com/wshobson/agents --skill vision-sft
Please help me install this Agent Skill. Skill: vision-sft Source: https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/vision-sft Command: npx skills add https://github.com/wshobson/agents --skill vision-sft
Fine-tuning a vision-language model often fails silently: the loss curve looks normal while the model learns nothing, due to image-placeholder mismatches or wrong resolution budgets. This Skill provides a validated adapter configuration recipe so VLM fine-tunes actually learn.
Ask the agent to configure a LoRA fine-tune for your vision-language model on your image-text dataset and validate the data collator before training.
High-intent search queries and answers about installing and using this skill.
Freeze the vision tower and projector, then apply LoRA to the LLM only across all linear layers (q/k/v/o and gate/up/down projections) at rank 8-16 with alpha 16-32. This is the settled default for adapting VLM behavior without disturbing visual representations.
Unfreeze only for genuine visual domain shift, such as satellite imagery or medical scans, after the frozen-tower recipe plateaus. Unfreeze at most the last six ViT layers and run the vision learning rate 5-10x lower than the LLM rate.
Two silent killers cause this: image placeholder tokens not mapping 1:1 to media items passed to the collator, and a min_pixels/max_pixels budget that downsamples images below task requirements. Run the pre-training validation checklist on one collated batch before relaunching.
No. QLoRA is permitted only with a frozen vision tower; quantizing the base while training vision layers is unsupported and unstable. If the vision tower must unfreeze, switch to bf16 LoRA instead.
No. When fast_inference=True, you must set finetune_vision_layers=False because vLLM cannot serve LoRA adapters on vision layers. Choose either unfrozen vision layers or fast inference, not both.
Monjyu1101Compute image-text similarity scores and top-k matches using CLIP.
jsdfhasuhGuide end-to-end LLM fine-tuning with dataset preparation, PEFT selection, and evaluation.
TubaSidExplain Vision-Language Model architecture, training, and construction steps.
Strategic-AutomationCombine supervised fine-tuning and reinforcement learning with TRL for model alignment.
