skills.rest
sp-202
sp-202COMMUNITY
Explore all skills by this author →
WebsiteGitHub

audit-comet-expression

Audit Comet Spark SQL expression implementations for correctness and test coverage gaps.

Updated Aug 27, 2026
Search Tags:#audit#rust#test coverage#datafusion#spark sql#expression parity#compatibility guide
One-click install
npx skills add https://github.com/sp-202/datafusion-comet-geo --skill audit-comet-expression-sp-202
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: audit-comet-expression
Source: https://github.com/sp-202/datafusion-comet-geo/tree/main/.claude/skills/audit-comet-expression
Command: npx skills add https://github.com/sp-202/datafusion-comet-geo --skill audit-comet-expression-sp-202

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you prevent silent correctness regressions by auditing whether a Comet expression behaves exactly like Spark across multiple Spark versions, and whether Comet has enough tests to prove that behavior.

Core Features & Use Cases

  • Cross-version Spark behavior review: Compares Spark 3.4.3, 3.5.8, and 4.0.1 implementation details for the target expression, including null handling, ANSI mode behavior, and edge cases.
  • Comet implementation and compatibility analysis: Checks the Comet Scala serde, any expression shims, and the Rust/DataFusion implementation for feature/support accuracy and error semantics.
  • Coverage gap analysis with actionable recommendations: Finds missing test dimensions by contrasting Spark test coverage against existing Comet SQL and Scala tests, then prioritizes what to add.
  • Documentation update: Updates the expression support documentation with version-specific findings so the compatibility guide stays accurate.

Quick Start

Ask an auditor to run the expression audit for $ARGUMENTS by cloning Spark v3.4.3, v3.5.8, and v4.0.1, collecting the relevant Spark and Comet test cases, and then producing a gap report with prioritized test recommendations.

Frequently Asked Questions about audit-comet-expression

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I audit a Comet expression for Spark SQL parity across multiple Spark versions?▼

To audit Comet expression parity, compare the Spark 3.4.3, 3.5.8, and 4.0.1 implementations against the Comet Scala serde and Rust/DataFusion execution paths to identify behavioral differences and missing test coverage.

What is expression parity testing in Comet and why is it needed?▼

Expression parity testing verifies that Comet's Rust/DataFusion execution of a Spark SQL expression matches Spark's native behavior, preventing silent correctness regressions across different Spark versions and ANSI mode edge cases.

How do I find missing test coverage for Spark SQL expressions in Comet?▼

Find missing test coverage by contrasting Spark's test cases against existing Comet SQL and Scala tests, generating a structured coverage gap matrix that prioritizes concrete new tests to add for expression parity.

Does Comet support the same null handling and ANSI mode behavior as Spark SQL expressions?▼

Comet audits check whether null handling and ANSI mode behavior match across Spark 3.4.3, 3.5.8, and 4.0.1, mapping behavior differences to Comet support metadata to verify compatibility and error semantics.

How do I update Comet documentation after auditing Spark SQL expression compatibility?▼

Update the expression support documentation by recording version-specific findings from the Spark and Comet compatibility analysis, ensuring the compatibility guide accurately reflects the audited Rust/DataFusion implementation behavior.

Can I use Comet to check Scala serde and Rust execution paths for Spark expression compatibility?▼

Yes, Comet audits check the Scala serde, expression shims, and Rust/DataFusion implementation to verify feature accuracy and error semantics, ensuring the Comet expression matches Spark SQL behavior across versions.

Related Skills

Explore More Software Engineering Skills→
apacheapache
1.2k

review-comet-pr

Guide structured review of DataFusion Comet PRs for Spark compatibility.

Official
Intermediate
manutejmanutej
61

apache-spark-data-processing

Teach Apache Spark data processing across RDDs, DataFrames, Spark SQL, streaming, and MLlib.

Community
Advanced
AnveioAnveio
1

role-code-reviewer

Review pull requests for correctness, architecture, and maintainability.

Community
Intermediate
apacheapache
1.2k

audit-comet-expression

Audit Comet expressions for correctness and test coverage across Spark versions.

Official
Advanced