Compares Model A and Model B outputs on Marlin V3 across multiple turns with traceable eval-state.
npx skills add https://github.com/MinhOmega/marlin-skill --skill eval-minhomega
Please help me install this Agent Skill. Skill: eval Source: https://github.com/MinhOmega/marlin-skill/tree/main/.claude/skills/eval Command: npx skills add https://github.com/MinhOmega/marlin-skill --skill eval-minhomega
This skill provides an end-to-end framework for evaluating competing model outputs (Model A and Model B) on Marlin V3 code across multiple turns, ensuring structured comparison, traceability, and reproducibility.
Start an evaluation session by invoking /eval with repo, PR, and turn parameters to begin recording turns.
High-intent search queries and answers about installing and using this skill.
Structured A/B evaluation compares Model A and Model B outputs across multiple turns using axis-based scoring. It maintains a reproducible eval-state with trace reading and baseline reference files for auditable cross-model comparisons.
Turn-based code evaluation analyzes model outputs sequentially across 13 axes, reading injected traces to score and justify responses. It maintains a cacheable context and eval-state to ensure strict reproducibility throughout the session.
Start an A/B evaluation session by invoking the eval command with repo, PR, and turn parameters. This initializes the eval-state and begins recording the independent A and B evaluation paths for comparison.
Yes, validating prompt engineering changes is a primary use case. The framework compares code review suggestions across model iterations, surfacing regressions by evaluating independent A and B paths with traceable, axis-based scoring.
The framework satisfies strict requirements for reproducibility, traceability, and guardrails in cross-model evaluation. It uses baseline reference files and eval-state management to ensure all A/B pair analysis remains fully auditable.
Axis-based scoring across 13 dimensions requires baseline reference files and injected traces to function correctly. Without proper eval-state management and cacheable context, maintaining strict reproducibility across multiple turns becomes challenging.
cescrafliAutomate Claude Code session evaluations with code, model, and human graders.
syncfusionBuild Angular stock charts with OHLC data and Syncfusion indicators.
thinking-machines-labCapture training metrics, logtrees, and traces into metrics.jsonl and logtree.json files.
pistosminCompare live page navigation and resource-timing data to baseline JSONs for PR regressions.
