[SPARK-52233][SQL] Fix map_zip_with for Floating Point Types

[mihailom-db]

What changes were proposed in this pull request?

Fix to map_zip_with expression while handling floating point numbers.

Why are the changes needed?

Previously we would run getKeysWithIndexesFast which would use LinkedHashMap, which does not use proper equality on keys for floating point numbers. All NaNs would be treated in a different way. This PR aims to fix this behaviour.

Example:

select map_zip_with(map(float('NaN'), 1), map(float('NaN'), 2), (k, v1, v2) -> (v1, v2))

Output before:

{"NaN":{"v1":1,"v2":null},"NaN":{"v1":null,"v2":2}}

Output after:

{"NaN":{"v1":1,"v2":2}}

Does this PR introduce any user-facing change?

Yes, fixing the way expression works.

How was this patch tested?

Added tests to golden files.

Was this patch authored or co-authored using generative AI tooling?

No.

[cloud-fan]

shall we simply changge getKeysWithIndexesFast to normalize the float/double values before putting into the hashmap?

[mihailom-db]

Changed the code now, could you take one more look?

[cloud-fan]

My proposal is to simply do

val normalized = if (key == Float.NaN) {
  Float.NaN
} else if (key == Double.NaN) {
  Double.NaN
} else {
  key
}
hashMap.put(key, indexes)

[cloud-fan]

The idea is to use the same NaN instance, so that the hash map can work out of the box.

[mihailom-db]

The problem is the way we are constructing the indexes. We first input the value from first map, and then check if the value is in the map. The hash map is not using the proper equals on NaN values, so we get into the same problem.

[mihailom-db]

So either we abstract separate method for Float/Double or we improve getKeysWithIndexesFast in this way.

[cloud-fan]

we can also normalize the key before lookup?