Write statistics

[jonashaag]

It seems like no Parquet column statistics (like min/max value) are written by this tool.

Would be nice to write add an option to add statistics or even enable them by default.

[pacman82]

I assumed the parquet-rs crate would do so by default. Seems I am wrong. What is a good way to test statistics have been written?

[pacman82]

Note to self: https://docs.rs/parquet/latest/parquet/file/properties/struct.WriterProperties.html#method.statistics_enabled

Happy to change the default, still wonder how to test this, though.

[pacman82]

Extracting statistics from a file written with odbc2parquet worked for me. To do this I used a piece of Python code kindly supplied by Uwe on Stack Overflow:

import pyarrow.parquet as pq

pq_file = pq.ParquetFile("tmp.par")
# Get metadata for the i-th RowGroup
rg_meta = pq_file.metadata.row_group(0)
# Get the "max" statistic for the k-th column
max_of_col = rg_meta.column(0).statistics.max
print(max_of_col)

Now, I wonder, what makes you think odbc2parquet does not write statistics? Are the statistics I saw calculated on demand?

[jonashaag]

Interestingly this seems to be the case with a single specific table only:

I can reproduce this by fetching a single column from a SQL Server database.

I'll try to create a reproducer for you next week.

[jonashaag]

Minimal reproducer on my SQL Server instance:

CREATE TABLE tmp (x CHAR(1))
INSERT INTO tmp (x) VALUES ('a')

Fetch with SELECT * FROM tmp.

[pacman82]

I can reproduce this, but I wonder if this is an issue. It parquet-rs seems to generate statistics for e.g. integer columns, but not for text. Explicitly enabling writing statistics in the writer properties does not change it. So if this is an issue, I think I would need to send it upstream. I wonder however if I am missing another flag I could set or if this could even be considered sensible behavior for text columns.

[pacman82]

Am I also right in assuming it is specifically about the min/max statistic?

[jonashaag]

Yes. This is important because it helps query engines skip row groups that are irrelevant.

Am I right to assume that this is the implementation used by this tool? I can open a ticket. https://github.com/apache/arrow-rs

[pacman82]

Am I right to assume that this is the implementation used by this tool? I can open a ticket. https://github.com/apache/arrow-rs

Yes this is the correct repository to raise an issue. odbc2parquet uses the parquet crate. The repository in question also is home to the arrow crate (which you might have guessed).

In case it could be of help, here is the point in odbc2parquets code there the writer properties are instantiated:

https://github.com/pacman82/odbc2parquet/blob/8415a466da726ccbca6dadf7102238ab74d59534/src/query/parquet_writer.rs#L48C10-L48C28

I rely very much on the defaults, but setting to produce statistics explicitly did not change behavior.

[pacman82]

Feel free to link this issue to the upstream ticket.

[jonashaag]

I wonder if it's the same as apache/arrow-rs#5162

[pacman82]

This might very well be the case. I think text is written as "physical" type bytes with logical type UTF-8

[jonashaag]

The ticket was unrelated. My ticket: apache/arrow-rs#5270

[pacman82]

This behavior is fixed in odbc2parquet 4.1.2 as it uses parquet 50.