-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: skip_workgroup_check setting to reduce AWS throttling #713
Conversation
@amacal implementation looks good. Please have a look at the ci, precommit checks are failing. Nice to have, and not mandatory, add some "integration tests" to fully test the feature in a real environment. |
@amacal plese remember what I suggested before:
Without all these 3 conditions, we cannot merge. Thanks |
I added a unit test and documented new parameter in the documentation. I cannot do functional test, because I cannot push it via org AWS account. |
@svdimchenko @Jrmyy could you have a look when you have some time? I'm currently away from keyboard. |
@amacal implementation looks fine, have a look at the CI, there is a failure |
Is there anything I need to do to have it merged? |
@amacal you changes looks good. As you can see repository ownership is being moved to dbtlabs, they are the official owners. |
Description
The adapter performs a GetWorkGroup operation on each thread, which is later cached. When dbt build is started for 10 independent models, it issues 10 AWS requests to get the same information. If dbt is orchestrated via Airflow to run 32 dbt tasks concurrently, it sends 320 GetWorkGroup requests, causing throttling on the AWS side.
This PR introduces a skip_workgroup_check setting, which instructs dbt to skip checking if a WorkGroup contains an enforced output location.
Checklist