implement-comet-expression

What problem does it solve?

Implementing a new Spark expression in DataFusion Comet is complex because it requires matching Spark’s exact semantics, wiring the serde and native execution layers, and proving correctness against SQL tests and an audit comparison to multiple Spark versions.

Core Features & Use Cases

• Spark-first semantics study: Locates the canonical Spark master implementation and its tests to understand input types, null handling, ANSI branches, and guards.
• Upstream reuse when possible: Checks the datafusion-spark crate for an existing implementation to avoid re-implementing behavior that already matches Spark.
• Full Comet wiring workflow: Guides adding the Scala serde, registering the protobuf mapping, creating the native physical function, and validating via Comet SQL tests plus an audit-driven iteration loop.
• Use Case: You want Comet to support a missing Spark SQL expression end-to-end so Spark SQL/DataFrame queries can run with Comet acceleration without semantic drift.

Quick Start

Use the implement-comet-expression skill with the target Spark expression name you want to add, so it produces a step-by-step plan from Spark source study through serde/native wiring, SQL test creation, and audit-driven fixes.