Allow reading Parquet maps that lack a values field
#6730
Conversation
There was a problem hiding this comment.
This looks good to me @etseidl -- thanky ou
I also poked around in https://github.com/apache/parquet-testing/tree/master/data for an example of such a file, but it seems like we do not have one.
| let out = reader.next().unwrap().unwrap(); | ||
| assert_eq!(out.num_rows(), 3); | ||
| assert_eq!(out.num_columns(), 2); | ||
| // map and list columns should now be equivalent |
There was a problem hiding this comment.
So this means the parquet reader will read MAP without values as an ListArray -- I think this seems like the intention so 👍
There was a problem hiding this comment.
Yes, because an arrow map must have non-null values IIUC.
Correct, which is why the effort in the tests to produce one. I'll try submitting one I have on hand to parquet-testing. |
|
apache/parquet-testing#63 submitted. Maybe we can hold off on merging this to see if the test file will be accepted. |
|
Looks like the test is in the final phases of being accepted. Let's wait for that to complete and then we can merge this one too |
|
Should the tests here be modified first to read the new file? |
Yes pls |
|
Not sure what markdown is failing in CI. Edit: I managed a conda env with prettier and there were no issues reported. Very puzzling (esp since there's no output beyond "this job failed"). Maybe it's just a transient thing. 🤷 |
I retriggered the job -- I bet it was some sort of github infrastructure blip |
Which issue does this PR close?
Closes #1642.
Rationale for this change
The Parquet spec does not require the
valuesfield of a map to be present, but current readers will error out if this field is missing.What changes are included in this PR?
Changes both the record reader and arrow reader to read a
MAPlackingvaluesas a list of keys. This matches the behavior of arrow-cpp.Are there any user-facing changes?
No