What problem does it solve?
Optimizing a Rust scalar expression in Apache DataFusion Comet without a disciplined process leads to fake wins: baselines measured against already-modified code, hidden regressions on dense-null or long-value shapes, and output that silently diverges from main. This Skill enforces a rigorous benchmark-first, no-regression workflow for performance work in native/spark-expr/.
Core Features & Use Cases
- Baseline-first benchmarking: Captures a criterion baseline from unmodified source before any edit, covering no-null, sparse-null, dense-null, short/long, valid/invalid, and ASCII/non-ASCII shapes.
- Bit-identical correctness gate: Requires unit tests to pass unchanged and output (values, null buffer, errors) to match main exactly before any submission.
- No-regression gate with noise filtering: Blocks PRs with reproducible regressions on any shape while teaching how to distinguish cross-run criterion noise from real effects via second samples.
- Use Case: You want to speed up a string kernel in native/spark-expr/. The Skill walks you through reading the optimizing_expressions.md guide, saving a main baseline, applying a technique from the catalog, re-measuring against the baseline, and recording a dated performance audit before opening a perf PR.
Quick Start
Optimize the native substring expression in the datafusion-comet spark-expr crate following the benchmark-first workflow.
