Handle filesystem latency when creating RUN_COMPLETE file
#803
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
I really don't like throwing an exception here.
I think we should try harder to force a cache update. Instead of just doing an exists, you might try tossing in a path.parent.iterdir(). Apparently opening the parent directory should cause the directory cache to refresh.
If that doesn't seem to be working, drop a symlink to the temp file location where you were going to move the original file, and walk away instead of throwing an exception.
Also, CREATION_ERROR is explicitly for test creation problems. I'd create a new 'WARNING' label for cases like this.
File system quirks mean that the RUN_COMPLETE file may not yet exist on the file system after it is created. Since the file is used by the same process relatively soon after creation, this can occassionally cause a FileNotFoundError. It was therefore necessary to wait for the file to be synced to the file system, and to impose a timeout. The previous timeout value was too short and resulted in TimeoutErrors relatively frequently. This commit increases the timeout value from 2 seconds to 30 seconds, per the suggestion of Paul Ferrell.
The idiosyncrocies of NFS means that when the RUN_COMPLETE file is written, it may not appear on the local file system, which results in a FileNotFoundError when that file is accessed. This commit adds a fallback when the file does not exist on the local file system, in the process reverting the previous solution which involved waiting and eventually raising a timeout. The fallback consists of the following steps: 1. Create the file as normal, and check whether it exists. 2. If it does not exist, attempt to force an NFS cache reset by listing the contents of the parent directory. 3. If the file still does not exist, create a symlink to the location where the file is expected eventually to be.
hwikle-lanl
force-pushed
the
802-uncaught-exception-when-creation-of-run_complete-file-times-out
branch
from
25f6f34 to
f7b6152
Compare
hwikle-lanl
changed the title
RUN_COMPLETE file
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Code review checklist: