Replace ScalarBuffer in Parquet with Vec (#1849) (#5177) #5178
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tustvold
commented
Dec 6, 2023
| @@ -339,8 +332,8 @@ where | |||
| Some(keys) => { | |||
| // Happy path - can just copy keys | |||
| // Keys will be validated on conversion to arrow | |||
| let keys_slice = keys.spare_capacity_mut(range.start + len); | |||
There was a problem hiding this comment.
This was actually incorrect, but didn't matter as spare_capacity_mut didn't update the length of the buffer directly, and so this would just potentially allocate more space than necessary
tustvold
commented
Dec 6, 2023
| use arrow_data::ArrayDataBuilder; | ||
| use arrow_schema::{DataType as ArrowType, TimeUnit}; | ||
| use std::any::Any; | ||
| use std::sync::Arc; | ||
|
|
||
| /// Provides conversion from `Vec<T>` to `Buffer` | ||
| pub trait IntoBuffer { |
There was a problem hiding this comment.
This whole module is crate private, so this isn't a breaking change
tustvold
commented
Dec 6, 2023
| /// | ||
| /// [scalar]: https://doc.rust-lang.org/book/ch03-02-data-types.html#scalar-types | ||
| /// | ||
| pub trait ScalarValue: Copy {} |
There was a problem hiding this comment.
This module is crate-private and so this is not a breaking change
tustvold
commented
Dec 6, 2023
| /// instead a subsequent call should be made to [`BufferQueue::set_len`] | ||
| fn spare_capacity_mut(&mut self, batch_size: usize) -> &mut Self::Slice; | ||
| /// instead a subsequent call should be made to [`BufferQueue::truncate_buffer`] | ||
| fn get_output_slice(&mut self, batch_size: usize) -> &mut Self::Slice; |
There was a problem hiding this comment.
I opted to rename these methods so they didn't collide with methods on Vec. Eventually the plan with #5178 is to remove the need for this trait entirely
|
As an added bonus this appears to yield some non-trivial performance improvements |
7 tasks
Dandandan
approved these changes
Dec 8, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #1849
Part of #5177
Rationale for this change
Following #3756 it is possible to construct arrow arrays directly from
Vecwithout copying. This PR updates parquet to do this, not only reducing the amount of code, but opening the door to pushingVecintoColumnReaderproper (#5177)What changes are included in this PR?
Are there any user-facing changes?