skills.rest
sp-202
sp-202COMMUNITY
Explore all skills by this author →
WebsiteGitHub

implement-comet-expression

Implement Spark SQL expressions in DataFusion Comet with serde and native execution wiring.

Updated Aug 27, 2026
Search Tags:#rust#datafusion-comet#spark expression#scala serde#sql tests#semantic parity
One-click install
npx skills add https://github.com/sp-202/datafusion-comet-geo --skill implement-comet-expression
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: implement-comet-expression
Source: https://github.com/sp-202/datafusion-comet-geo/tree/main/.claude/skills/implement-comet-expression
Command: npx skills add https://github.com/sp-202/datafusion-comet-geo --skill implement-comet-expression

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Implementing a new Spark expression in DataFusion Comet is complex because it requires matching Spark’s exact semantics, wiring the serde and native execution layers, and proving correctness against SQL tests and an audit comparison to multiple Spark versions.

Core Features & Use Cases

  • Spark-first semantics study: Locates the canonical Spark master implementation and its tests to understand input types, null handling, ANSI branches, and guards.
  • Upstream reuse when possible: Checks the datafusion-spark crate for an existing implementation to avoid re-implementing behavior that already matches Spark.
  • Full Comet wiring workflow: Guides adding the Scala serde, registering the protobuf mapping, creating the native physical function, and validating via Comet SQL tests plus an audit-driven iteration loop.
  • Use Case: You want Comet to support a missing Spark SQL expression end-to-end so Spark SQL/DataFrame queries can run with Comet acceleration without semantic drift.

Quick Start

Use the implement-comet-expression skill with the target Spark expression name you want to add, so it produces a step-by-step plan from Spark source study through serde/native wiring, SQL test creation, and audit-driven fixes.

Frequently Asked Questions about implement-comet-expression

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add a missing Spark expression to DataFusion Comet?▼

Adding a Spark expression to DataFusion Comet requires studying Spark master semantics, wiring the Scala serde and Rust native execution layers, and validating correctness through Comet SQL file tests and an audit loop against multiple Spark versions.

What is the process for ensuring semantic parity when implementing Spark SQL expressions in Comet?▼

Ensuring semantic parity involves studying the canonical Spark master implementation for input types and null handling, checking the datafusion-spark crate for existing logic, and iteratively adding audit-recommended tests until support is acceptable.

How do I validate Spark expression support in Comet against multiple Spark versions?▼

You validate Spark expression support in Comet by running Comet SQL file tests and executing the audit-comet-expression tool, then iteratively adding audit-recommended tests until the expression's behavior matches Spark semantics acceptably.

Can I reuse existing Spark expression logic from datafusion-spark when adding support in Comet?▼

Yes, you can reuse existing logic by checking the datafusion-spark crate for an existing implementation that already matches Spark behavior, avoiding the need to re-implement the expression semantics from scratch in the Rust native execution layer.

What steps are required to wire a new Spark expression through the Comet serde and native execution layers?▼

Wiring a new Spark expression requires adding the Scala serde, registering the protobuf mapping, creating the native physical function, and validating the end-to-end integration through Comet SQL tests and an audit-driven iteration loop.

Why do Comet SQL tests fail when adding new Spark expressions and how do I fix them?▼

Comet SQL tests fail when the native Rust execution drifts from Spark semantics, requiring you to run the audit-comet-expression tool and iteratively add audit-recommended tests to resolve behavioral mismatches and achieve acceptable support.

Related Skills

Explore More Software Engineering Skills→
manutejmanutej
61

apache-spark-data-processing

Teach Apache Spark data processing across RDDs, DataFrames, Spark SQL, streaming, and MLlib.

Community
Advanced
huseyindenizhuseyindeniz
15

test-audit

Analyze test and Storybook coverage gaps with structured reports.

Community
Advanced
AnveioAnveio
1

role-code-reviewer

Review pull requests for correctness, architecture, and maintainability.

Community
Intermediate
apacheapache
1.2k

audit-comet-expression

Audit Comet expressions for correctness and test coverage across Spark versions.

Official
Advanced