Incorrect stats getting populated for a Decimal column while converting DeltaTable to IcebergTable using XTable

[Prajwaltr011]

Search before asking

• I had searched in the issues and found no similar issues.

Please describe the bug 🐞

Recently, we attempted to use Xtable on our database to convert Delta Tables to Iceberg Tables using the Xtable jar. The conversion was successful. However, we encountered discrepancies when reading the statistics of the Iceberg table. Specifically, for the decimal columns, the upper and lower bounds were incorrectly calculated in the Iceberg Avro snapshots file compared to native Iceberg tables. For instance, if the minimum and maximum of a decimal column are -8.0 and -5.0, the stats are showing 0.8 and 0.5, which is completely incorrect.
call_center.zip

Are you willing to submit PR?

• I am willing to submit a PR!
• I am willing to submit a PR but need help getting started!

Code of Conduct

• I agree to follow this project's Code of Conduct

[the-other-tim-brown]

@Prajwaltr011 can you attach the _delta_log? The stats are copied directly from the delta log so it will be easier to reproduce this case with that information.

[Prajwaltr011]

Sure @the-other-tim-brown. Please find attached delta logs files. One important thing to note it down is that we have ran "explain analyze" command on these tables using Trino which calculated trino meta stats. I am not sure whether this will have any impact on Iceberg stats population.

00000000000000000000.json
00000000000000000001.json
extended_stats.json

[the-other-tim-brown]

Thanks @Prajwaltr011 I think there is some issue in how we create the intermediate representation of column stats data from the stats json in Delta Lake. I am working on some improvements to our integration tests to see if I can reproduce this.

[the-other-tim-brown]

I hit some other issues while setting up the integration test but have confirmed that the Iceberg stats are stored by calling .unscaledValue() on the BigDecimal and the scale is not properly set on the intermediate value when extracting the decimal from Delta causing the shift in the decimal point in Iceberg.

[Prajwaltr011]

Thank you for catching this! and hope to see the fix soon.

[the-other-tim-brown]

I found the issue can also happen for Hudi sources so patched that as well here: #617

[Prajwaltr011]

Fantastic work!! @the-other-tim-brown