Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(runner): Fix broadcast in LocalRunner #11923

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kagamiori
Copy link
Contributor

Summary: This diff adds a unit test of LocalRunner with a broadcast node in the query plan and fixes the handling of broadcast in LocalRunner.

Differential Revision: D67549347

Summary: This diff adds a unit test of LocalRunner with a broadcast node in the query plan and fixes the handling of broadcast in LocalRunner.

Differential Revision: D67549347
@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Dec 21, 2024
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D67549347

Copy link

netlify bot commented Dec 21, 2024

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit f8f0044
🔍 Latest deploy log https://app.netlify.com/sites/meta-velox/deploys/676605bffc62ea0008a2bb57

@kagamiori kagamiori requested a review from xiaoxmeng December 21, 2024 00:03
Copy link
Contributor

@xiaoxmeng xiaoxmeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kagamiori looks good overall % comments. Thanks!

VELOX_CHECK_NOT_NULL(output);
const auto producerPrefix = current_->taskPrefix;
auto result = planNode_;
newFragment();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leave an empty line here.

root_->stack_.pop_back();
auto* consumer = root_->stack_.back();
fragments_.back().numBroadcastDestinations = consumer->current_->width;
VELOX_CHECK_GE(consumer->current_->width, 1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should check width before its use, and we shall also check numBroadcastDestinations is not set in the previous fragment yet.

}
exchange(output->outputType(), VectorSerde::Kind::kPresto);
auto* exchange = dynamic_cast<const core::ExchangeNode*>(planNode_.get());
consumer->current_->inputStages.push_back(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a non null check on exchange after cast

@@ -108,6 +108,29 @@ core::PlanNodePtr DistributedPlanBuilder::shuffleResult(
return std::move(planNode_);
}

core::PlanNodePtr DistributedPlanBuilder::broadcastResult() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we call this shuffleBroadcastResult? And rename the existing one to shufflePartitionedResult?

const auto producerPrefix = current_->taskPrefix;
auto result = planNode_;
newFragment();
root_->stack_.pop_back();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we check if stack is empty on dtor? And we should expect stack is not empty when calling non-result APIs from DistributedPlanBuilder? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants