[SPARK-52267] Match field ID in ParquetToSparkSchemaConverter

[chenhao-db]

What changes were proposed in this pull request?

In the vectorized Parquet reader, there are two classes to resolve the Parquet schema when reading a Parquet file:

• ParquetReadSupport: it clips the Parquet schema to only include the necessary part used by the Spark requested schema. The matching considers both field name and ID.
• ParquetToSparkSchemaConverter: it resolves the Parquet schema to a Spark type by connecting it to the Spark requested schema. The matching only considers field name.

When the field ID matches but field name doesn't, the first step will clip the Parquet schema to the same structure as the Spark requested schema as expected. In the second step, the Parquet type cannot be connected to a Spark type in the requested schema, and it will be inferred as a Spark type. It will usually work as expected if the inferred type is the same as the requested type. But it is possible that they are different and the read is still valid. For example, if the Parquet type is int and the Spark type is long. In this case, the vectorized Parquet reader will produce int data in column vectors, which will be interpreted as long data by subsequent operations.

This can happen in real user cases if an Iceberg table with both rename and change column type (int -> long) operations is converted into a Delta table. This situation may be very rare, though.

This PR fixes by bug by matching field ID in ParquetToSparkSchemaConverter when the name cannot be matched. I know that ParquetReadSupport gives priority to field ID when it exists, but I am not fully confident about this change and would like to keep the semantic change minimal.

Why are the changes needed?

It fixes a correctness issue.

Does this PR introduce any user-facing change?

Yes, as stated above.

How was this patch tested?

Unit test.

Was this patch authored or co-authored using generative AI tooling?

No.

[chenhao-db]

@cloud-fan @jackierwzhang Please take a look. Thanks!