Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Select of Blink Table should be Flat #4469

Open
cpwright opened this issue Sep 8, 2023 · 4 comments
Open

Select of Blink Table should be Flat #4469

cpwright opened this issue Sep 8, 2023 · 4 comments
Assignees
Labels
core Core development tasks feature request New feature or request query engine triage
Milestone

Comments

@cpwright
Copy link
Contributor

cpwright commented Sep 8, 2023

As a user, I want select on a blink table to produce a flat table so that the select will have memory usage bounded by the largest update rather than the number of row keys that it sees over time.

@cpwright cpwright added feature request New feature or request triage labels Sep 8, 2023
@nbauernfeind
Copy link
Member

Maybe we should create a blink-table supported version of flatten which redirects column sources using the added rowset. We could invoke the flatten on the source table and then defer the select on that sub-table. Same idea, but maybe provides a better encapsulation of logic.

@cpwright
Copy link
Contributor Author

cpwright commented Sep 8, 2023

If you flatten before select, it should have the same effect; and select() could throw that into the preamble in io.deephaven.engine.table.impl.QueryTable#selectInternal that already flattens for many static selects. The flatten should convert all adds/removes to adds and removes from 0...n-1.

From a user-perspective, however, we should ideally never have to think about our address space and flattening it, when the engine can do it for us without loss of performance.

@cpwright
Copy link
Contributor Author

cpwright commented Sep 8, 2023

I said that it was equivalent, the result is equivalent; but the performance is a little worse as the select operation would need to read through a redirection instead of just writing out to flat (but still growable) sources.

@rcaudy rcaudy added this to the Backlog milestone Sep 8, 2023
@rcaudy rcaudy added query engine core Core development tasks labels Sep 8, 2023
@rcaudy rcaudy assigned nbauernfeind and unassigned rcaudy Sep 19, 2023
@rcaudy
Copy link
Member

rcaudy commented Sep 19, 2023

Do we want the same flattening for append-only? I think yes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core development tasks feature request New feature or request query engine triage
Projects
None yet
Development

No branches or pull requests

3 participants