You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Given a limited memory budget, one may want to customize buffer size for different column chunks based on their actual size, i.e., smaller chunks will use consume less memory budget for its buffer.
Jinpeng Zhou / @jp0317:
I think we can close this one for now as it may not worth making all these changes for a certain scenario. I'll revisit this if it became more favorable. Thanks for all the comments and reviews.
When buffered stream is enabled, all column chunks, regardless of their actual sizes, are currently sharing the same buffer size which is stored in the shared [read properties](https://github.com/apache/arrow/blob/main/cpp/src/parquet/file_reader.cc#L213).
Given a limited memory budget, one may want to customize buffer size for different column chunks based on their actual size, i.e., smaller chunks will use consume less memory budget for its buffer.
Reporter: Jinpeng Zhou / @jp0317
PRs and other links:
Note: This issue was originally created as PARQUET-2321. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: