Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Fix avg_bytes_per_rows when node 0 does not have data #4

Open
wants to merge 2 commits into
base: feature/refactor-distribution-will
Choose a base branch
from

Conversation

Christian8491
Copy link

No description provided.

total_avg_bytes_per_row.push_back(avg_bytes_per_row);

// let's recompute the avg_bytes_per_row if node master does not have data
if (avg_bytes_per_row == 1) {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should always use that info, not only if the master node does not have it.
Second, i think you should weight the avg_bytes_per_row info by the amount of data from that node, such that if one node has little data but has a very high or low avg_bytes_per_row, it does not skew the results. So I would recommend doing something like:

size_t total_bytes = 0; 
size_t total_rows = 0;
for(std::size_t i = 0; i < nodes.size(); ++i) {
total_bytes += total_avg_bytes_per_row[i]*total_table_rows[i];
total_rows += total_table_rows[i];
}
size_t final_avg_bytes_per_row = total_bytes/total_rows;


Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and use final_avg_bytes_per_row for generate_partition_plan

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated as requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants