fix: canonicalize NaN in flat arrays_overlap float keys

[sunchao]

Why are the changes needed?

arrays_overlap answers a simple question: do two arrays contain at least one common value? For floating-point arrays, Spark treats every representation of NaN as the same value, even when their underlying IEEE 754 sign bits or payloads differ. Comet's native implementation compared those raw bit patterns instead. As a result, enabling Comet could silently change the answer to an otherwise ordinary Spark SQL query.

Consider a Parquet table containing a NaN:

CREATE TABLE t_flat(x DOUBLE) USING parquet;
INSERT INTO t_flat VALUES (DOUBLE('NaN')), (0.0D), (1.0D);

SELECT x,
       arrays_overlap(array(x), array(-x)) AS overlap
FROM t_flat;

For the first row, -x is still a NaN, but negating it at execution time changes its sign bit. Spark correctly considers that value equal to the original NaN. On Spark 3.4–4.1, before this change, Comet incorrectly treated the two representations as different:

Input x	Spark 3.4–4.1	Comet before	Comet after
NaN	true	false	true
+0.0	false	false	false
1.0	false	false	false

The +0.0 row is also a compatibility control: -x becomes -0.0, and Spark 3.4–4.1 considers the two zeros different for flat arrays. Fixing NaN comparisons must not change that result.

Spark 4.2 and later intentionally behave differently. Under SPARK-54918, Spark's optimizer normalizes both floating-point arrays_overlap operands before evaluation. Consequently, +0.0 and -0.0 overlap, and both Spark and Comet return true. The native comparison must still preserve distinct zero keys for earlier Spark versions; on Spark 4.2+, the operands have already been normalized before those keys are constructed.

On Spark 3.4–4.1, the mismatch also affects SQL null semantics. If the first array additionally contains NULL, Spark returns true because the two NaN values overlap, while the previous Comet implementation returns NULL because it misses that definite match:

SELECT arrays_overlap(
  array(x, CAST(NULL AS DOUBLE)),
  array(-x)
)
FROM t_flat
WHERE isnan(x);

-- Spark: true
-- Comet before: NULL
-- Comet after: true

Runtime negation in these examples is intentional. Parquet canonicalizes floating-point NaN values when writing them, so storing a negative or custom-payload NaN in the input file is not enough to reproduce the bug. A distinct representation must be produced after the scan.

The signed-zero boundary is therefore version-specific:

SELECT arrays_overlap(array(0.0D), array(-0.0D));

-- Spark 3.4–4.1 and Comet: false
-- Spark 4.2+ and Comet: true (operands are normalized first)

A general-purpose floating-point normalizer that also merges signed zero inside Comet's comparison key would fix the NaN mismatch but introduce a Spark 3.4–4.1 compatibility regression.

Closes #5270.

What changes were proposed in this PR?

The change gives Comet's existing flat-array comparison the same floating-point equality contract as Spark. When Comet derives a comparison key for a FLOAT or DOUBLE, every NaN representation is mapped to one canonical key. All other values retain their original bit patterns. This preserves distinct signed zeros when Spark 3.4–4.1 passes them through unchanged, while respecting Spark 4.2's earlier normalization of the operands.

Applying the correction at the shared comparison-key boundary fixes both ways the existing implementation checks for overlap: direct comparisons for small arrays and hash-based lookups for larger arrays. The execution plan, fast-path structure, null propagation, and behavior for non-floating-point values are otherwise unchanged.

This PR intentionally does not change nested-array comparisons. Spark uses a different equality contract for nested values, and the separate nested signed-zero issue is addressed in #5235.

Mixed scalar/array argument broadcasting is another pre-existing issue, tracked separately in #5269. This change does not alter that argument-dispatch behavior.

How was this PR tested?

The native Rust regressions construct positive, negative, and signaling NaN bit patterns directly for both floating-point widths. They verify NaN equivalence, distinct signed-zero keys, SQL null handling, and both sides of the small-array/hash-lookup threshold. All 23 focused arrays_overlap tests pass.

Most integration coverage now lives in the existing expressions/array/arrays_overlap.sql fixture, as requested in review. Its Parquet-backed queries generate noncanonical NaNs after the scan and exercise FLOAT and DOUBLE, reversed operands, nullable arrays, signed-zero controls, and arrays large enough to use hash lookup. The SQL-file harness compares Comet with the running Spark version and checks native execution, so it also covers Spark 4.2's different signed-zero behavior without duplicating version-specific expected results.

A small Scala test remains to inspect the actual floating-point bits. It confirms that Parquet canonicalizes stored NaNs and that native runtime negation produces noncanonical NaNs after the scan—something SQL equality alone cannot establish.

Both focused integration tests pass locally on Spark 3.5, Spark 4.0, and Spark 4.2. Maven Spotless checks also pass for all three profiles.

Native formatting, compilation, and focused tests, from native/:

cargo fmt --all -- --check
cargo build
cargo test -p datafusion-comet-spark-expr --lib \
  array_funcs::arrays_overlap::tests -- --nocapture

The focused Spark commands were run from the repository root with JDK 17 configured. Clean reactor builds were used when switching profiles:

./mvnw -B -ntp test -Pspark-3.5 -Dtest=none \
  '-Dsuites=org.apache.comet.CometSqlFileTestSuite expressions/array/arrays_overlap.sql' \
  -Dscalastyle.skip=true
./mvnw -B -ntp test -Pspark-3.5 -Dtest=none \
  '-Dsuites=org.apache.comet.CometArrayExpressionSuite runtime NaN representations' \
  -Dscalastyle.skip=true

./mvnw -B -ntp clean test -Pspark-4.0 -Dtest=none \
  '-Dsuites=org.apache.comet.CometSqlFileTestSuite expressions/array/arrays_overlap.sql' \
  -Dscalastyle.skip=true
./mvnw -B -ntp test -Pspark-4.0 -Dtest=none \
  '-Dsuites=org.apache.comet.CometArrayExpressionSuite runtime NaN representations' \
  -Dscalastyle.skip=true

./mvnw -B -ntp clean test -Pspark-4.2 -Dtest=none \
  '-Dsuites=org.apache.comet.CometSqlFileTestSuite expressions/array/arrays_overlap.sql' \
  -Dscalastyle.skip=true
./mvnw -B -ntp test -Pspark-4.2 -Dtest=none \
  '-Dsuites=org.apache.comet.CometArrayExpressionSuite runtime NaN representations' \
  -Dscalastyle.skip=true

[comphead]

Thanks @sunchao looks solid to me, would be great to have tests mostly in sql file

src/test/resources/sql-tests/expressions/array/arrays_overlap.sql

[sunchao]

Thanks @comphead! Moved the behavioral cases into the existing arrays_overlap.sql in 247e0b1. The SQL suite now compares against the running Spark version and covers runtime-generated NaNs, signed zeros, nulls, reversed operands, and both the short-array and hash-lookup paths for FLOAT and DOUBLE.

I kept a small Scala raw-bit test because SQL equality cannot distinguish NaN representations. It verifies that runtime negation produces noncanonical NaNs after Parquet has canonicalized the stored values.

Both focused integration tests pass locally on Spark 3.5, 4.0, and 4.2, and all 23 targeted Rust tests pass.

[comphead]

Thanks @sunchao