Skip to content

Conversation

hendrikmakait
Copy link
Contributor

@hendrikmakait hendrikmakait commented Feb 9, 2024

Adjusts default cluster sizes following changes suggested in #1166.

Changes:
Scale 10

  • Reduced number of VMs by half from 16 to 8 machines. This is probably better as it felt opverprovisioned.
  • Halfed single VM size from m6i.8xlarge (32 CPUs, 128 GiB RAM) to m6i.4xlarge (16 CPUs, 64 GiB RAM) to match.

Scale 100
No changes

Scale 1000

  • Doubled VM size form m6i.large (2 CPUs, 8 GiB RAM) to m6i.xlarge (4 CPUs, 16 GiB RAM).
    During our latest runs, we've noticed that the configuration for scale 1000 is not large enough and needs to be doubled.
  • Doubled single VM size from m6i.16xlarge (64 CPUs, 256 GiB RAM) to m6i.32xlarge (128 CPUs, 256 GiB RAM) to match.

Scale 10000

  • Reduced number of VMs by half from 64 to 32 to match single VM size.
    Note: We have not run this yet but this is the only way to keep CPU count constant as there's no larger m6i VM available for the single VM types.

@phofl
Copy link
Contributor

phofl commented Feb 9, 2024

I think the argument for scale 10 000 was that this is one of the features of distributed frameworks that you can scale out

@hendrikmakait
Copy link
Contributor Author

I think the argument for scale 10 000 was that this is one of the features of distributed frameworks that you can scale out

That makes sense, but then we should, at the very least, highlight that it's not a fair comparison.

elif scale == 1000:
return {
"worker_vm_types": ["m6i.large"],
"worker_vm_types": ["m6i.xlarge"],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noted that pyspark query 21 would always fail with this setup, but will succeed with m6i.2xlarge.
Maybe that's okay though? If we succeed with this and pyspark mostly succeeds w/ other queries except one, then too bad for them(?).

Copy link
Contributor

@milesgranger milesgranger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @hendrikmakait.

@hendrikmakait hendrikmakait merged commit e2935b0 into main Feb 12, 2024
@hendrikmakait hendrikmakait deleted the adjust-default-clusters-for-tpch branch February 12, 2024 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants