compare-experiments

What problem does it solve?

When working with multiple completed open-xquant experiment runs, manually cross-referencing strategy specs, backtest metrics, audit results, and equity curves is time-consuming and error-prone. This Skill automates the comparison process to quickly surface meaningful differences between runs, helping you understand how strategy changes impact performance without manual file parsing.

Core Features & Use Cases

• Automated Spec Diffing: Recursively compares strategy specification files between two runs to highlight parameter, logic, and configuration changes, along with their likely impact on performance.
• Validated Metric Comparison: Only compares runs that have completed all required audits and backtest steps, ensuring metric differences are based on comparable, audited data. Generates side-by-side metric deltas for returns, Sharpe ratio, and drawdown.
• Visual Comparison Outputs: Creates overlay visualizations of equity curves and drawdowns, plus a bar chart of key metric differences, to make performance gaps easy to interpret.
• Use Case: For example, if you modified a strategy's risk management rule and ran two backtests, use this Skill to quickly see if the change improved returns, reduced drawdown, or introduced unintended side effects, while confirming both runs passed all audit checks.

Quick Start

Use the compare-experiments skill to compare your two completed open-xquant experiment runs in the runs/ directory and generate a full comparison report with performance visualizations and spec difference analysis.