-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
File containing a Map schema without explicitly required key #47
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
@@ -50,6 +50,7 @@ | |||||||||
| float16_zeros_and_nans.parquet | Float16 (logical type) column with NaNs and zeros as min/max values. . See [note](#float16-files) below | | ||||||||||
| concatenated_gzip_members.parquet | 513 UINT64 numbers compressed using 2 concatenated gzip members in a single data page | | ||||||||||
| byte_stream_split.zstd.parquet | Standard normals with `BYTE_STREAM_SPLIT` encoding. See [note](#byte-stream-split) below | | ||||||||||
| hive-map-schema.parquet | Contains a Map schema without explicitly required keys, produced by Presto. See [note](#hive-map-schema) | | ||||||||||
|
||||||||||
TODO: Document what each file is in the table above. | ||||||||||
|
||||||||||
|
@@ -387,3 +388,40 @@ To check conformance of a `BYTE_STREAM_SPLIT` decoder, read each | |||||||||
`BYTE_STREAM_SPLIT`-encoded column and compare the decoded values against | ||||||||||
the values from the corresponding `PLAIN`-encoded column. The values should | ||||||||||
be equal. | ||||||||||
|
||||||||||
## Hive Map Schema | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why "Hive"? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This was due to schema being labeled with |
||||||||||
|
||||||||||
A number of producers, such as Presto/Trino/Athena, create files with schemas where the Map fields are not explicitly marked as required. An optional key is not possible according to the Parquet spec, but the schema is getting created this way. | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
||||||||||
This issue has been fixed in: | ||||||||||
- [Trino v386+](https://github.com/trinodb/trino/commit/3247bd2e64d7422bd13e805cd67cfca3fa8ba520) | ||||||||||
- [Presto v0.274+](https://github.com/prestodb/presto/commit/842b46972c11534a7729d0a18e3abc5347922d1a) | ||||||||||
|
||||||||||
Of course it will take some time for all new files to be produced with these fixes, and the amount of existing data out there remains. | ||||||||||
|
||||||||||
We can recreate these problematic files for testing [arrow-rs #5630](https://github.com/apache/arrow-rs/pull/5630) with relevant Presto/Trino CLI, or with AWS Athena Console: | ||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||
|
||||||||||
```sql | ||||||||||
CREATE TABLE my_catalog.my_table_name WITH (format = 'Parquet') AS ( | ||||||||||
SELECT MAP ( | ||||||||||
ARRAY['name', 'parent'], | ||||||||||
ARRAY[ | ||||||||||
'report', | ||||||||||
'another' | ||||||||||
] | ||||||||||
) my_map | ||||||||||
) | ||||||||||
``` | ||||||||||
|
||||||||||
The schema in the created file is: | ||||||||||
|
||||||||||
``` | ||||||||||
message hive_schema { | ||||||||||
OPTIONAL group my_map (MAP) { | ||||||||||
REPEATED group key_value (MAP_KEY_VALUE) { | ||||||||||
OPTIONAL BYTE_ARRAY key (STRING); | ||||||||||
OPTIONAL BYTE_ARRAY value (STRING); | ||||||||||
} | ||||||||||
} | ||||||||||
} | ||||||||||
``` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of "hive", can we name this e.g. "incorrect_map_schema.parquet"?