Statistics truncate length is not applied in DataFusion compaction

[patchwork01]

Description

In a compaction on DataFusion, we set the statistics truncate length in the DataFusion ParquetOptions under max_statistics_size. This property is not actually used.

Steps to reproduce

1. Set a statistics truncate length in the table property sleeper.table.parquet.statistics.truncate.length
2. Ingest data that will result in statistics longer than this truncate length
3. Run a compaction in DataFusion
4. See the statistics in the file output by the compaction are not truncated, although it was in the input file

Expected behaviour

The statistics truncate length should be applied in DataFusion.

Background

DataFusion has deprecated the configuration option datafusion.execution.parquet.max_statistics_size, because it's not used:

• Deprecate max statistics size properly apache/datafusion#14188
• Deprecate "max statistics size" property in WriterProperties apache/arrow-rs#6884

It seems to have been replaced in the Parquet library by statistics_truncate_length, added here:

• Enable truncation of binary statistics columns apache/arrow-rs#5076

There doesn't seem to be a way to set this in DataFusion. We've raised this issue to add an option to configure it:

• Configure statistics_truncate_length in Parquet writer apache/datafusion#14601