Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add retry factory to consolidate retry strategies across dbt-bigquery #1395

Merged
merged 43 commits into from
Nov 20, 2024
Merged

Add retry factory to consolidate retry strategies across dbt-bigquery #1395

merged 43 commits into from
Nov 20, 2024
+713 −715

Conversation

mikealfare
Copy link
Contributor

Problem

We handle retries in different ways across different classes and layers of the application. We regularly get bug reports related to retries not being properly handled.

Solution

Dedicate a class specifically to building retry strategies and remove this logic from the execution code.

Checklist

  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

@mikealfare mikealfare self-assigned this Nov 5, 2024
@cla-bot cla-bot bot added the cla:yes label Nov 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 5, 2024

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-bigquery contributing guide.

@mikealfare
reorder this tuple to make the pr review easier to understand
3f79642
@mikealfare
move client factory to credentials module so that on_error can be mov…
f300080 
…ed to the retry factory in the retry module
@mikealfare
move on_error factory to retry module
c3065e5
@mikealfare
move client factories from python_submissions module to credentials m…
ad74114 
…odule
@mikealfare
create a clients module
9029c49
@mikealfare
retry all client factories by default
bc0fbea
@mikealfare
move polling from manual check in python_submissions module into retr…
9a9f87e 
…y_factory
@mikealfare
move load_dataframe logic from adapter to connection manager, use the…
136ea77 
… built-in timeout argument instead of a manual polling method
@mikealfare
move upload_file logic from adapter to connection manager, use the bu…
90d5308 
…ilt-in timeout argument instead of a manual polling method, remove the manual polling method
@mikealfare
move the retry to polling for done instead of create
9211e1c
@mikealfare
fix broken import in tests from code migration
e90c24d
@mikealfare
align new retries with original methods, simplify retry factory
a2db35b
@mikealfare
fix seed load result
b8408c2
@mikealfare
create a method for the dataproc endpoint
5b896ee
@mikealfare
add some readability updates
43c10f1
@mikealfare
add some readability updates
4256682
@mikealfare
add some readability updates, simplify submit methods
5644509
@mikealfare
make imports explicit, remove unused constant
df2971b
@mikealfare mikealfare enabled auto-merge (squash) November 20, 2024 04:27
@mikealfare mikealfare merged commit 83bb413 into main Nov 20, 2024
38 checks passed
@mikealfare mikealfare deleted the add-retry-factory branch November 20, 2024 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@colin-rogers-dbt colin-rogers-dbt colin-rogers-dbt approved these changes

@VersusFacit VersusFacit VersusFacit approved these changes

Assignees

@mikealfare mikealfare

Labels
cla:yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mikealfare @VersusFacit @colin-rogers-dbt