What problem does it solve?
Deciding which Spark expressions running through Comet's JVM codegen dispatcher are worth reimplementing natively in Rust is error-prone and repetitive. This Skill scores candidates on Spark compatibility confidence and native performance upside, records verdicts in a persistent audit log so disqualified expressions are never re-litigated, and files exactly one well-structured GitHub issue per run.
Core Features & Use Cases
- Candidate Selection: Builds the expression pool from Comet's serde definitions (not the docs table), excludes ruled-out families, already-assessed expressions, and ones with existing issues, then ranks by workload presence in TPC-H/TPC-DS.
- Two-Axis Scoring: Rates compatibility confidence against a hazard checklist (JVM formatting APIs, regex features, collation, BigDecimal rounding, timezones) and native upside based on per-row allocation, vectorization techniques, and Spark fallback cases.
- Empirical Calibration: When a rating is borderline, gathers real evidence by inspecting generated kernel code, running proxy benchmarks, or measuring allocation rates.
- Audit Trail and Issue Filing: Appends a dated verdict line to per-category audit pages under docs/source/contributor-guide/expression-audits/ and files a labeled GitHub issue with the full assessment.
- Use Case: A maintainer asks which codegen-dispatched expression to tackle next; the Skill surveys the pool, picks the top candidate, verifies no prior issue exists, scores it, and files a ready-to-pick-up proposal issue.
Quick Start
Run the suggest-native-expression skill with an expression name like 'mask' to assess it, or with no argument to survey the whole pool and recommend the best native candidate.
