Rewrite execution of microbatch models to avoid blocking the main thread #11332
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…stration to a runner We're working to ensure the orchestration of microbatch batches doesn't block the main thread. This will require a lot of disentangling that currently exists in run.py. As such, it made sense to "quickly" stub out a guide of what needs to be done.
The `MicrobatchBatchRunner` will be for running individual batches, whereas the `MicrobatchModelRunner` will handle the orchestration of the batches to be run for a given model.
…Runner` directly Previously `handle_job_queue` considered `MicrobatchModelRunner` special cases, and delegated to `handle_microbatch_model` to orchestrate the batches instead of delegating to the `MicrobatchModelRunner` directly. Now that the `MicrobatchModelRunner` will be handling batch orchestration, we can appropriately delegate to it, and remove the special casing.
The function won't work as is, but I felt it better to straight copy, commit, and then modify it to work in the runner context iteratively.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #11332 +/- ##
==========================================
- Coverage 88.97% 82.92% -6.05%
==========================================
Files 189 189
Lines 24182 24172 -10
==========================================
- Hits 21516 20045 -1471
- Misses 2666 4127 +1461
Flags with carried forward coverage won't be shown. Click here to find out more.
|
We don't need these functions in `MicrobatchModelRunner` because the inherited versions of these methods from `ModelRunner` will work for our needs. Of note, we can probably also remove the need of having these functions in `MicrobatchBatchRunner` by renaming the `print_batch_start_line` and `print_batch_result_line` to the method names that the `ModelRunner` methods call.
…unner` The `MicrobatchModelRunner.compile` does nothing because `MicrobatchModelRunner` only orchestrates the batches of the model to run, and doesn't actually run the sql of the model. Thus compilation is unnecessary in `MicrobatchModelRunner`
Of note, implementing `on_skip` for `MicrobatchModelRunner` is unecessary because the inherited `on_skip` suffices.
This is necessary because the materialization executor needs the `MicrobatchBuilder` in order to build the jinja context.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #11243
Problem
Solution
Checklist