wire-datafusion-function

To wire a DataFusion function into Comet, you select the correct integration pattern, update the QueryPlanSerde.scala file with the appropriate CometScalarFunction mapping, and add Spark SQL file tests mirroring upstream input types and edge cases.

Comet preserves Spark semantics by matching return types, null handling, ANSI error paths, input type restrictions, and foldability rules during the serde wiring process to ensure the native function adheres to the Spark contract.

You need to register a native UDF in Comet specifically when using the datafusion-spark pattern B integration path, whereas other wiring patterns may only require a passthrough or custom serde mapping without explicit UDF registration.

The best way to test compatibility is to add Spark SQL file tests that exercise validity bitmaps, edge cases like NaN and infinities, boundary values, and fallback behavior when the Scala serde layer returns None.

Wired DataFusion functions fail Spark SQL tests in Comet when null handling, ANSI behavior, or foldability rules do not match the Spark contract, or when the incorrect serde wiring pattern was selected for the specific expression requirements.

You can use Comet to integrate DataFusion functions with Spark expressions only when the upstream function already exists, requiring you to select the lightest correct integration approach to match Spark contract details accurately.