Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[BUG] Bump Parquet reader max_page_size to 256MB (#1553)
Bumps the max_page_size to `usize::MAX` Note that it appears parquet2 uses this argument for BOTH the max **page header** thrift structure size, and also the **page data** size itself. This is different from how other libraries do it. Here we set it to 256MB: 1. This is a very large number that should capture most corner-cases 2. If anything is larger than this number there is most likely an issue with the Parquet file itself that should be addressed 3. Reading in 256MB of page data at once is still unlikely to OOM a system so we should be very safe there Closes: #1551 --------- Co-authored-by: Jay Chia <[email protected]@users.noreply.github.com>
- Loading branch information