catalog.add_parquet_partitions doesn't update other statistics e.g. ObjectCount, recordCount, sizeKey on catalog

[kp-commit]

Hi there,

I was trying out the catalog.add_parquet_partitions method, which is pretty neat. I noticed that it doesn't update the Catalog's other properties automatically that seem to be important in query statistics e.g. ObjectCount, recordCount, sizeKey. Can this be added? It would be nice to have.

Regards,
Kirti

[jaidisido]

Hi @kp-commit,

The add_parquet_partitions method implements the create_batch_partition call, which simply creates one more partitions in batch. The statistics that you are referring to (e.g. ObjectCount) are Glue table level stats and not partition level. For instance, creating a new partition would not affect the number of records in the table since it's not creating any new data, just updating the metadata related to partitions.