Parquet error: Invalid offset in sparse column chunk data

[b4l]

Describe the bug

Querying on a registered parquet table sometimes throws an error. The query is doing a range query over multiple fields. The same query with different literals, in which only one range impacts the output as the others are unbound works fine. Also, this happens for SQL and DataFrame alike.

To Reproduce

No response

Expected behavior

Query returns result.

Additional context

No response

[tustvold]

This should have been fixed by apache/arrow-rs#5036

Perhaps you could try out #8029

[b4l]

@tustvold thanks! It is indeed working as expected with #8029. Looking forward to the release.

[alamb]

FWIW we have released arrow 48.0.1 (see apache/arrow-rs#5050) so a cargo update in your project should get the fix @b4l

[alamb]

Please let us know if that doesn't work

[liukun4515]

We should add some comment to this issue or the related pr.

If user don't want to update the arrow-rs or the datafusion version to resolve this issue, they can disable the page index:
set the enable_page_index to false. to skip this bug.

cc @alamb @tustvold

[alamb]

If user don't want to update the arrow-rs or the datafusion version to resolve this issue, they can disable the page index:
set the enable_page_index to false. to skip this bug.

That is a good point

The workaround suggested by @liukun4515 can be implemented like this:

SQL

❯ set datafusion.execution.parquet.enable_page_index = false;
0 rows in set. Query took 0.000 seconds.

Programatically

By setting ParquetOptions::enable_page_index to false