job-manager: fix prolog/epilog exception handling #5321
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: The job-manager 'perilog' plugin terminates any running prolog if the job receives an exception. This works as expected, but the code doesn't check whether a running process is the prolog or epilog, so job epilogs are killed on job exception as well. This isn't desired behavior since this doesn't allow the epilog to properly complete, and also causes the nodes on which the epilog is running to be drained. Ensure that the running process is a prolog before killing it. Fixes flux-framework#5319
Problem: The perilog plugin terminates a running prolog on any job exception, but it should only do so for a fatal job exception. Check the job exception severity in the exception event callback and only terminate the prolog for exceptions with severity=0.
Problem: No tests in the testsuite ensure that a nonfatal exception during the job prolog does not terminate the prolog. Add a test to check this expected behavior.
|
Thanks! Setting MWP. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes a couple issues in the handling of exceptions in the
perilogplugin:This will require flux-framework/flux-sched#1047 to be merged so that the flux-sched checks won't fail.