Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable spilling support for partial aggregation #7930

Closed
zhztheplayer opened this issue Dec 8, 2023 · 1 comment
Closed

Enable spilling support for partial aggregation #7930

zhztheplayer opened this issue Dec 8, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@zhztheplayer
Copy link
Contributor

zhztheplayer commented Dec 8, 2023

Description

For more earlier discussion, refer to #7511

Partial aggregation is expected to reduce data size transmitted from mapper to reducer. So far Velox's partial aggregation flushes intermediate data out once memory usage reaches limit, so more or less the data reduction ratio will be subject to the frequency of flushing.

This issue would introduce the possibility of brining spilling support to partial aggregation, to make it be able to maintain the lowest data reduction ratio despites the memory resource limit. This could be useful when:

  1. When reducer number is far less than mapper number
  2. When network between mappers and reducers is bad
  3. When shuffle/exchange cost between mappers and reducers is high

In the fix, we'd introduce an option which is by default false to enable spilling support for partial aggregation. When the option is true, disable flushing at the same time.

@zhztheplayer
Copy link
Contributor Author

Closing as not planned. See background here.

@zhztheplayer zhztheplayer closed this as not planned Won't fix, can't repro, duplicate, stale Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant