Write parquet column chunk file_offset #15882
Conversation
pmattione-nvidia
added
libcudf
|
CC @etseidl, the resident Parquet writer wizard |
| auto& column_chunk_meta = row_group.columns[i].meta_data; | ||
| auto const chunk_offset = _current_chunk_offset[p]; | ||
| auto& column_chunk = row_group.columns[i]; | ||
| column_chunk.file_offset = chunk_offset; |
There was a problem hiding this comment.
Danger Will Robinson! There appears to be confusion in the community about what file_offset is. According to the parquet thrift this points to the chunk metadata, not the start of the chunk. This allows for the metadata to be separated from the thrift in the footer, both to allow for faster parsing, and to support data living in a file separate from where the footer is located. (See apache/parquet-format#242 (comment) for some discussion of the latter use case). It seems as though parquet-mr incorrectly populates this with the offset of the first page, while arrow does what the spec mandates and uses it to point to a duplicate copy of the ColumnMetaData that is located at the end of a column chunk (as per the figure in the spec). Since libcudf does not write this copy, it's best to leave this field as '0'.
There was a problem hiding this comment.
Ah I see. OK, I'll close this PR.
Set the file_offset member of the parquet column chunk metadata, which was not set before.
Checklist