-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[native] Derive TableScan stream type as FIXED #24468
[native] Derive TableScan stream type as FIXED #24468
Conversation
@arhimondr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Is there a reason why only native execution creates fixed number of drivers? |
In Presto (Java) the number of TableScan drivers created was chosen adaptively. That added a lot of complexity to TaskExecutor and the benefit was not clear. In Velox the number of TableScan drivers is fixed. The adaptivity can still be achieved via delivering splits to only a subset of TableScan drivers without going into complexity of managing the number of drivers dynamically. Here's an example where @xiaoxmeng adds TableScan scaling: facebookincubator/velox#11879 |
320464c
to
f45a1eb
Compare
Ready for review |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @arhimondr
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @arhimondr for this PR and the context.
In native a fixed number of drivers is created for a pipeline containing TableScan
f45a1eb
to
0ba7548
Compare
@arhimondr has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Description
In native a fixed number of drivers is created for a pipeline containing TableScan
Motivation and Context
To avoid unnecessary local exchange when fixed number of streams is necessary (for example for TableWriteOperator)
Impact
Improved performance for certain type of INSERT INTO queries
Test Plan
CI, Unit Tests
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.