Buffering object_store requests across column chunks

[ilsley]

I am accessing a 3-column parquet file using ParquetRecordBatchStream<ParquetObjectReader> and AmazonS3 from object_store. As far as I can tell, reads are buffered successfully across pages to reduce round trips, although each query might require three round trips, one for each column chunk. In this particular parquet file, the first two columns compress very well, so the get requests are relatively near each other. For example, get_request in S3Client is called as follows:

column 1: 4..234
column 2: 277..1645
column 3: 1691..196969

Firstly, are these gets buffered elsewhere, or does this result in three round trips? If so, is there a way to collapse these requests into a single one? Most requests would typically fall within a single row group, so it would also work to read the whole row group in this case.

[tustvold]

If using https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#method.get_ranges which is what https://docs.rs/parquet/latest/parquet/arrow/async_reader/struct.ParquetObjectReader.html does for each row group, then it will coalesce gaps smaller than https://docs.rs/object_store/latest/object_store/constant.OBJECT_STORE_COALESCE_DEFAULT.html

[ilsley]

Thank you, that helps. I have used the ArrowReaderBuilder like this:

let parquet_reader = reader_builder
  .with_row_groups(rg_indices)
  .with_projection(projection)
  .with_row_filter(row_filter)
  .build()?;

The row_filter predicates use values in the first two columns. So, I think what's happening is that read_row_group in ReaderFactory<AsyncFileReader> is calling row_group.fetch twice in order to refine which rows are needed. The third fetch is then to get the correct rows in the third column chunk.

If this is the case, do you think it might be better to have smaller row groups, and filter based on row groups rather than using with_row_filter?

[tustvold]

Have you enabled reading and writing the page index?

[ilsley]

Yes, the page index is enabled like this:

let options = ArrowReaderOptions::new().with_page_index(true);
let metadata = ArrowReaderMetadata::load_async(&mut reader, options).await?;

The builder is created with ParquetRecordBatchStreamBuilder::new_with_metadata.

---

It looks to me like the presence of the row filter causes the following line in the parquet crate to run, once for each predicate:
https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/async_reader/mod.rs#L467C1-L469C29
This fetches data from the first two column chunks, which is what I'm hoping to avoid.

Removing the filter, .with_row_filter(row_filter), leads to a single fetch even with the projection mask including columns 2 and 3.
I think it could work if I refine the initial fetch using row_selection based on page statistics with some fine-grained post-fetch filtering.

[tustvold]

Yes, you will want to use those in combination