GH-31992: [C++][Parquet] Handling the special case when DataPageV2 values buffer is empty #45252
Conversation
|
|
github-actions
bot
added
Component: Parquet
Component: C++
awaiting review
github-actions
bot
added
awaiting committer review
|
|
||
| ASSERT_EQ(100, this->metadata_num_values()); | ||
| this->ReadColumn(Compression::SNAPPY); | ||
| ASSERT_EQ(0, this->values_read_); |
There was a problem hiding this comment.
I've verified that the unittest test would test this case
| page_buffer->data() + levels_byte_len, | ||
| uncompressed_len - levels_byte_len, | ||
| decompression_buffer_->mutable_data() + levels_byte_len)); | ||
| // GH-31992: DataPageV2 may store only levels and no values |
There was a problem hiding this comment.
Can we fix this in the Snappy decompressor instead?
There was a problem hiding this comment.
Yes, which one do you prefer?
There was a problem hiding this comment.
It depends whether an empty buffer is a normal compression result or a shortcut taken by parquet-java. Let me see.
There was a problem hiding this comment.
Ok, it's really a bug in parquet-java, because a 0-size buffer compressed to a 1-size buffer using Snappy.
There was a problem hiding this comment.
So should I handle the 0,0 case in snappy 🤔?
There was a problem hiding this comment.
So should I handle the
0,0case in snappy 🤔?
No, sorry. We should work around it in Parquet.
There was a problem hiding this comment.
Yes, I mean parquet-java is able to produce 0-sized compressed data as the example. How can I reproduce your case where 0 input is compressed to 1 byte?
There was a problem hiding this comment.
It's not my case, it's just what Snappy produces when you ask it to compress 0 byte.
There was a problem hiding this comment.
it's just what Snappy produces when you ask it to compress 0 byte.
IMO that's:
- "Compressed" Data is 0 byte
- Actually, the levels holds k bytes ?
I don't know how parquet-java works, I tried parquet rust and it failed to read
There was a problem hiding this comment.
Again, a 0-byte compressed data is invalid. It's probably a special case in the Parquet Java implementation.
|
@pitrou so what should I need to do to merge this? Should I check #45252 (comment) in compression to check that "Compress" doesn't compress the zero-sized page? Or I should change other code? |
|
@mapleFU Can we open a bug for Parquet-java and reference it here? |
|
Created: apache/parquet-java#3122 |
Rationale for this change
In DataPageV2, the levels and data will not be compressed together. So, we might get the "empty" data page buffer.
When meeting this, Snappy C++ will failed to decompress the
(input_len == 0, output_len == 0)data.What changes are included in this PR?
Handling the case in
column_reader.ccAre these changes tested?
Are there any user-facing changes?
Minor fix