Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarify num-nulls handling in Statistics and ColumnIndex #449

Merged
merged 9 commits into from
Aug 23, 2024
Merged

Clarify num-nulls handling in Statistics and ColumnIndex #449

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
dbfe60f
Clarify num-nulls handling
mapleFU Aug 16, 2024
0ebe97f
Update src/main/thrift/parquet.thrift
mapleFU Aug 19, 2024
d3f210f
Add readers behavior
mapleFU Aug 21, 2024
2e0df3a
Apply suggestions from code review
mapleFU Aug 21, 2024
fe815ea
Reader: change SHOULD to MUST
mapleFU Aug 21, 2024
1508aa2
Add for ColumnIndex
mapleFU Aug 21, 2024
0bf50c5
Apply suggestions from code review
mapleFU Aug 21, 2024
4dc4502
Update src/main/thrift/parquet.thrift
mapleFU Aug 21, 2024
aee6e1e
Update src/main/thrift/parquet.thrift
mapleFU Aug 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion src/main/thrift/parquet.thrift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,11 @@ struct Statistics {
*/
1: optional binary max;
2: optional binary min;
/** count of null value in the column */
/**
* count of null value in the column
mapleFU marked this conversation as resolved.
Show resolved Hide resolved
*
* Writers should write this field even if it is zero or in non-null columns.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Writers should write this field even if it is zero or in non-null columns.
* Writers SHOULD always write this field even if it is zero (a.k.a. no null value)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might also add the expectation to the reader implementations?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might also add the expectation to the reader implementations?

I'd like to wait other replying the context in ML firstly

*/
3: optional i64 null_count;
/** count of distinct values occurring */
4: optional i64 distinct_count;
Expand Down