Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-38071: [C++][CI] Fix Overlap column chunk ranges for pre-buffer #38073

Merged
merged 5 commits into from
Oct 8, 2023
Merged

GH-38071: [C++][CI] Fix Overlap column chunk ranges for pre-buffer #38073

merged 5 commits into from
Oct 8, 2023

Conversation

mapleFU
Copy link
Member

@mapleFU mapleFU commented Oct 6, 2023 •
edited
Loading

Rationale for this change

The C++ Parquet Arrow fuzz will generate bad Parquet file with bad row-range, this patch change the CoalesceReadRanges to return Result<>.

What changes are included in this PR?

Just a checking, change CoalesceReadRanges to return Result<>.

Are these changes tested?

No.

Are there any user-facing changes?

No.

@github-actions
Copy link

github-actions bot commented Oct 6, 2023

⚠️ GitHub issue #38071 has been automatically assigned in GitHub to PR creator.

@mapleFU mapleFU requested a review from pitrou October 6, 2023 12:26
@mapleFU mapleFU changed the title GH-38071: [C++] Fix Overlap column chunk ranges for pre-buffer GH-38071: [C++][CI] Fix Overlap column chunk ranges for pre-buffer Oct 6, 2023
@github-actions github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 6, 2023
@mapleFU
Copy link
Member Author

mapleFU commented Oct 6, 2023

@jorisvandenbossche @lidavidm I've add an Result<> as return value here

@github-actions github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Oct 6, 2023
mapleFU
mapleFU commented Oct 6, 2023
View reviewed changes
cpp/src/arrow/io/util_internal.h
std::vector<ReadRange> CoalesceReadRanges(std::vector<ReadRange> ranges,
int64_t hole_size_limit,
int64_t range_size_limit);
Result<std::vector<ReadRange>> CoalesceReadRanges(std::vector<ReadRange> ranges,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't know if this matter, since it change the public api.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's in an internal header so no expectation that it remains stable

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's good, thanks!

@mapleFU mapleFU marked this pull request as ready for review October 6, 2023 15:05
@mapleFU mapleFU requested a review from wgtmac as a code owner October 6, 2023 15:05
@mapleFU
Copy link
Member Author

mapleFU commented Oct 6, 2023 

[ RUN      ] AzureFileSystem.UploadThenDownload
unknown file: Failure
C++ exception with description "Fail to get a new connection for: http://127.0.0.1:10000./ Couldn't connect to server" thrown in the test body.
[  FAILED  ] AzureFileSystem.UploadThenDownload (6053 ms)

Hmm...Seems this is not caused by me 🤔

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Oct 6, 2023
kou
kou approved these changes Oct 7, 2023
View reviewed changes
Copy link
Member

@kou kou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

The Azure failure will be unrelated. I re-ran the job.

@github-actions github-actions bot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels Oct 7, 2023
@mapleFU
Copy link
Member Author

mapleFU commented Oct 8, 2023

I've rebase the master to rerun the test :-) Hope this time it passes...

@kou
Copy link
Member

kou commented Oct 8, 2023

Passed. :-)
I'll merge this.

@kou kou merged commit 0b9f817 into apache:main Oct 8, 2023
33 of 34 checks passed
@kou kou removed the awaiting merge Awaiting merge label Oct 8, 2023
@conbench-apache-arrow
Copy link

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 0b9f817.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

@mapleFU mapleFU deleted the parquet-dcheck-fix-on-lazy branch October 9, 2023 02:12
@jorisvandenbossche
Copy link
Member

Thanks a lot @mapleFU for fixing this!

JerAguilon pushed a commit to JerAguilon/arrow that referenced this pull request Oct 23, 2023
@mapleFU
apacheGH-38071: [C++][CI] Fix Overlap column chunk ranges for pre-buf…
6d92ab0 
…fer (apache#38073)

### Rationale for this change

The C++ Parquet Arrow fuzz will generate bad Parquet file with bad row-range, this patch change the `CoalesceReadRanges` to return `Result<>`.

### What changes are included in this PR?

Just a checking, change `CoalesceReadRanges` to return `Result<>`.

### Are these changes tested?

No.

### Are there any user-facing changes?

No.

* Closes: apache#38071

Authored-by: mwish <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
@jorisvandenbossche jorisvandenbossche mentioned this pull request Oct 24, 2023
Closed
loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
@mapleFU @loicalleyne
apacheGH-38071: [C++][CI] Fix Overlap column chunk ranges for pre-buf…
aadb69c 
…fer (apache#38073)

### Rationale for this change

The C++ Parquet Arrow fuzz will generate bad Parquet file with bad row-range, this patch change the `CoalesceReadRanges` to return `Result<>`.

### What changes are included in this PR?

Just a checking, change `CoalesceReadRanges` to return `Result<>`.

### Are these changes tested?

No.

### Are there any user-facing changes?

No.

* Closes: apache#38071

Authored-by: mwish <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
dgreiss pushed a commit to dgreiss/arrow that referenced this pull request Feb 19, 2024
@mapleFU @dgreiss
apacheGH-38071: [C++][CI] Fix Overlap column chunk ranges for pre-buf…
44d7e13 
…fer (apache#38073)

### Rationale for this change

The C++ Parquet Arrow fuzz will generate bad Parquet file with bad row-range, this patch change the `CoalesceReadRanges` to return `Result<>`.

### What changes are included in this PR?

Just a checking, change `CoalesceReadRanges` to return `Result<>`.

### Are these changes tested?

No.

### Are there any user-facing changes?

No.

* Closes: apache#38071

Authored-by: mwish <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lidavidm lidavidm lidavidm approved these changes

@kou kou kou approved these changes

@pitrou pitrou Awaiting requested review from pitrou

@wgtmac wgtmac Awaiting requested review from wgtmac wgtmac is a code owner

Assignees
No one assigned
Labels
Component: C++ Component: Parquet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[C++] Enabling CacheOptions::LazyDefault caused Parquet fuzzing failure
4 participants
@mapleFU @kou @jorisvandenbossche @lidavidm