trl-fine-tuning

Load a model with AutoModelForCausalLM, prepare a prompt-completion or conversational dataset, configure SFTConfig with output directory and learning rate, then pass everything to SFTTrainer and call train(). Enable packing for 2-3x faster training.

DPO aligns models directly on preference pairs without a reward model. RLOO uses a trained reward model for online RL and replaces the removed PPO in TRL 1.x. GRPO is the memory-efficient option that compares groups of completions using custom reward functions.

Yes, PPOTrainer, PPOConfig, and the ppo CLI script were removed in TRL 1.x. Use RLOOTrainer or the trl rloo command as the drop-in replacement for reward-model-driven RLHF, or GRPO for memory-efficient online RL.

For 7B models: SFT needs about 16GB with LoRA, DPO needs 24GB since it stores a reference model, RLOO needs 40GB for policy plus reward model, and GRPO needs about 24GB. Reduce memory with LoRA/QLoRA and gradient checkpointing.

Increasing loss is expected behavior in GRPO because the loss measures KL divergence from the initial policy as the model learns. Monitor reward metrics and reward_std instead; reward should rise while reward_std stays above 0.1 to avoid mode collapse.

Reduce per_device_train_batch_size to 1, lower max_length from 1024 to 512, and increase gradient_accumulation_steps to maintain effective batch size. You can also enable gradient checkpointing on the model to trade compute for memory.