skills.rest
wshobson
wshobsonCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

trace-to-training-data

Convert graded evaluation traces into SFT examples and DPO preference pairs.

39.3k|4.2k|Updated Jul 24, 2025
Search Tags:#fine-tuning#LLM evaluation#DPO#sft#training data#preference pairs#rejection sampling
One-click install
npx skills add https://github.com/wshobson/agents --skill trace-to-training-data
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: trace-to-training-data
Source: https://github.com/wshobson/agents/tree/main/plugins/llm-finetuning/skills/trace-to-training-data
Command: npx skills add https://github.com/wshobson/agents --skill trace-to-training-data

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Graded evaluation traces and production logs often sit unused after an eval run, even though they contain exactly the labeled examples needed for fine-tuning. This Skill converts those verdicts and rewards into training-ready SFT rows and DPO preference pairs without re-judging or hand-labeling.

Core Features & Use Cases

  • SFT Conversion: Keep the top-reward fraction of passing traces, route expert-corrected failures directly into the SFT set, and apply step-level loss masking for partially bad multi-step trajectories.
  • Preference Pair Construction: Build DPO pairs from passing-vs-failing trajectories on the same task, selecting the rejected member at μ−2σ of the reward distribution and filtering by judge-score delta.
  • Hygiene Enforcement: Scan for secrets and PII, hold out eval golden IDs from training data, dedup against the existing training set, and record run_id/trace_id provenance for the dataset card.
  • Use Case: After an eval harness grades 500 production traces with pass/fail verdicts and rewards, convert them into a deduplicated SFT dataset plus DPO pairs, with goldens held out and provenance logged, ready for the fine-tuning pipeline.

Quick Start

Convert the graded traces in runs/latest/results.json into SFT rows and DPO pairs, holding out any task IDs that appear in eval/goldens.jsonl.

Frequently Asked Questions about trace-to-training-data

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I turn evaluation traces into fine-tuning data?▼

Take graded traces carrying a verdict and reward, keep the top-reward fraction of passing trajectories as SFT messages rows, and build DPO pairs from passing versus failing traces on the same task. Strip grading metadata from training rows but record task_id and trace_id as provenance.

How do I build DPO preference pairs from model outputs?▼

Pair traces sharing the same task_id: the chosen member is the top-reward trace, and the rejected member is whichever trace sits closest to μ−2σ of that task's reward distribution, not the absolute minimum. Filter candidate pairs by chosen-minus-rejected judge delta to cut volume without losing signal.

Can I use failing traces for training data?▼

Yes, when a human expert corrects the failing output, that correction goes directly into the SFT set without a reward threshold. For multi-step traces where only some steps are bad, mask the loss on bad steps instead of discarding the whole trajectory.

Why must eval goldens be excluded from training data?▼

A trace that also appears as an eval golden trains the model on the exact item it gets graded against later, silently inflating every subsequent eval run. Run a holdout check matching task_id against goldens.jsonl before merging any converted batch.

What should I do with traces that have no verdict or reward?▼

A trace without a verdict or reward is not convertible yet. Route it back to the eval harness for grading rather than hand-labeling it during conversion, since re-judging inside the conversion step signals a missing grader upstream.

Related Skills

Explore More Software Engineering Skills→
dwmkerrdwmkerr
23

anthropic-evaluations

Design and evaluate AI agent tasks with Anthropic eval patterns.

Community
Basic
M4n5terM4n5ter
3

training-data-curation

Standardize LLM post-training data collection and evaluation across SFT, DPO, and RLHF workflows.

Community
Intermediate
devMoezdevMoez
1

fine-tuning-with-trl

Automates LLM post-training alignment with TRL using SFT, DPO, and PPO/GRPO workflows.

Community
Advanced
AndurilCodeAndurilCode
7

context-eval

Evaluate context engineering harnesses against no-harness baselines with graded assertions.

Official
Advanced