Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(arrow-select): concat kernel will merge dictionary values for list of dictionaries #6893

Merged
merged 16 commits into from
Jan 4, 2025
Merged

feat(arrow-select): concat kernel will merge dictionary values for list of dictionaries #6893

merged 16 commits into from
Jan 4, 2025

Conversation

rluvaton
Copy link
Contributor

@rluvaton rluvaton commented Dec 17, 2024 •
edited
Loading

Which issue does this PR close?

Closes #6888

What changes are included in this PR?

  1. Updated concat kernel to merge dictionary values for list of dictionary
  2. Created merge function in BufferOffset for merging multiple BuffersOffset into one (which needed for the concat merge)

Are there any user-facing changes?

Yes, the merge function in the BufferOffset struct

@rluvaton
feat(arrow-select): make list of dictionary merge dictionary keys
34ab0ab 
TODO:
- [ ] Add support to nested lists
- [ ] Add more tests
- [ ] Fix failing test
@rluvaton
fix concat lists of dictionaries
66825fb
@github-actions github-actions bot added the arrow Changes to the arrow crate label Dec 17, 2024
rluvaton
rluvaton commented Dec 17, 2024
View reviewed changes
arrow-select/src/concat.rs Outdated Show resolved Hide resolved
rluvaton
rluvaton commented Dec 17, 2024
View reviewed changes
arrow-select/src/concat.rs Outdated Show resolved Hide resolved
@rluvaton rluvaton changed the title feat(arrow-select): make list of dictionary merge dictionary keys feat(arrow-select): concat kernel will merge dictionary values for list of dictionaries Dec 17, 2024
rluvaton
rluvaton commented Dec 17, 2024
View reviewed changes
arrow-select/src/concat.rs Outdated Show resolved Hide resolved
tustvold
tustvold reviewed Dec 17, 2024
View reviewed changes
arrow-select/src/concat.rs Outdated
@@ -129,12 +130,106 @@ fn concat_dictionaries<K: ArrowDictionaryKeyType>(
Ok(Arc::new(array))
}

fn concat_list_of_dictionaries<OffsetSize: OffsetSizeTrait, K: ArrowDictionaryKeyType>(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would significantly reduce the codegen to instead split concatenating the dictionary values, from re-encoding the offsets. As an added bonus this could also be done recursively.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated, please let me know it that was what you meant, I think it will be slower this way, no?

tustvold
tustvold reviewed Dec 17, 2024
View reviewed changes
Copy link
Contributor

@tustvold tustvold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've taken a quick look and left some suggestions on how to simplify this, in particular separating the concatenation of the list values and the list offsets will result in less code, require less codegen, and likely perform better.

i.e. something like (not tested)


DataType::List(f) => {
    let values: Vec<_> = arrays.iter().map(|x| x.as_list::<i32>().values()).collect();
    let new_values = concat(&values)?;
    let offsets = OffsetBuffer::from_lengths(
        arrays.iter().flat_map(|x| x.as_list::<i32>().offsets()
             .windows(2).map(|w| w[1].as_usize() - w[0].as_usize()))
    );
    let nulls = ...
    ListArray::new(f, offsets, new_values, nulls)
}

@tustvold tustvold marked this pull request as draft December 18, 2024 10:51
@rluvaton rluvaton marked this pull request as ready for review December 21, 2024 16:39
tustvold
tustvold reviewed Dec 21, 2024
View reviewed changes
Copy link
Contributor

@tustvold tustvold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking very promising, just a small comment r.e. avoiding ArrayData

I'll try to find some time for a more thorough review in the next few days if nobody else gets there first

@tustvold tustvold merged commit 2799268 into apache:main Jan 4, 2025
26 checks passed
CurtHagenlocher pushed a commit to CurtHagenlocher/arrow-rs that referenced this pull request Jan 13, 2025
@rluvaton @CurtHagenlocher
feat(arrow-select): concat kernel will merge dictionary values for …
94a079f 
…list of dictionaries (apache#6893)

* feat(arrow-select): make list of dictionary merge dictionary keys

TODO:
- [ ] Add support to nested lists
- [ ] Add more tests
- [ ] Fix failing test

* fix concat lists of dictionaries

* format

* remove unused import

* improve test helper

* feat: add merge offset buffers into one

* format

* add reproduction tst

* recommit

* fix clippy

* fix clippy

* fix clippy

* improve offsets code according to code review

* use concat dictionaries

* add specialize code to concat lists to be able to use the concat dictionary logic

* remove the use of ArrayData
svencowart pushed a commit to elastiflow/arrow-rs that referenced this pull request Jan 14, 2025
@rluvaton @svencowart
feat(arrow-select): concat kernel will merge dictionary values for …
5946840 
…list of dictionaries (apache#6893)

* feat(arrow-select): make list of dictionary merge dictionary keys

TODO:
- [ ] Add support to nested lists
- [ ] Add more tests
- [ ] Fix failing test

* fix concat lists of dictionaries

* format

* remove unused import

* improve test helper

* feat: add merge offset buffers into one

* format

* add reproduction tst

* recommit

* fix clippy

* fix clippy

* fix clippy

* improve offsets code according to code review

* use concat dictionaries

* add specialize code to concat lists to be able to use the concat dictionary logic

* remove the use of ArrayData
totoroyyb pushed a commit to totoroyyb/arrow-rs that referenced this pull request Jan 20, 2025
@rluvaton @totoroyyb
feat(arrow-select): concat kernel will merge dictionary values for …
8234675 
…list of dictionaries (apache#6893)

* feat(arrow-select): make list of dictionary merge dictionary keys

TODO:
- [ ] Add support to nested lists
- [ ] Add more tests
- [ ] Fix failing test

* fix concat lists of dictionaries

* format

* remove unused import

* improve test helper

* feat: add merge offset buffers into one

* format

* add reproduction tst

* recommit

* fix clippy

* fix clippy

* fix clippy

* improve offsets code according to code review

* use concat dictionaries

* add specialize code to concat lists to be able to use the concat dictionary logic

* remove the use of ArrayData
This was referenced Jan 27, 2025
Closed
Draft
alamb
alamb reviewed Jan 27, 2025
View reviewed changes
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have been testing the arrow 54.1.0 downstream in DataFusion and it appears this PR has introduced a regression:

I will try and look at it tomorrow if no one beats me to it.

This was referenced Jan 27, 2025
Open
Closed
@alamb
Copy link
Contributor

alamb commented Jan 28, 2025

Bug is related to concatenating sliced list arrays

tustvold
tustvold reviewed Jan 28, 2025
View reviewed changes
arrow-select/src/concat.rs

let values: Vec<&dyn Array> = lists
.iter()
.map(|x| x.values().as_ref())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the bug is here, it needs to slice values() by the first offset

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe you are correct (also needs to slice values to get the last offset)
Proposed fix here:

@alamb alamb mentioned this pull request Jan 28, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tustvold tustvold tustvold left review comments

@alamb alamb alamb left review comments

Assignees
No one assigned
Labels
arrow Changes to the arrow crate
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

arrow::compute::concat should merge dictionary type when concatenating list of dictionaries
3 participants
@rluvaton @alamb @tustvold