Reduce maptask transitions between WaitingForResources and CheckingSubtaskExecutions #4790
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s the same Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
dosubot
bot
added
size:S
EngHabu
approved these changes
Jan 30, 2024
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #4790 +/- ##
==========================================
- Coverage 58.64% 58.57% -0.08%
==========================================
Files 590 624 +34
Lines 51473 53627 +2154
==========================================
+ Hits 30187 31411 +1224
- Misses 18861 19698 +837
- Partials 2425 2518 +93
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
pingsutw
approved these changes
Jan 30, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Tracking issue
NA
Why are the changes needed?
In maptask, if any subtasks are in the
PhaseWaitingForResourcesphase then the entire maptask task transitions to that phase. The issue is that thek8s-arrayplugin uses a transition throughPhaseLaunchto get back toPhaseCheckingSubtaskExecutionshere. This causes events to be sent to flyteadmin to satisfy the phase update. With this commit flyteadmin began processing phase reason updates in batches, which causes all reason updates to persist in a timeline. The continual transition betweenPhaseWaitingForResourcesandPhaseCheckingSubtaskExecutionscauses a large amount of duplicated reasons in the time series.What changes were proposed in this pull request?
This PR proposed to treat
PhaseWaitingForResourcesandPhaseCheckingSubtaskExecutionsthe same so that events are only sent when subtask phases are updated rather than duplicate transitions between these maptask phases.How was this patch tested?
Locally.
Setup process
Screenshots
Check all the applicable boxes
Related PRs
NA
Docs link
NA