feat: improve reliability & usability of VFolder removal API #1884
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
comp:manager
github-actions
bot
added
the
comp:storage-proxy
|
Need to limit the concurrent deletion jobs for each vfolder to one and the maximum concurrent number of vfolder deletion jobs for each storage volume (configurable, defaults to 20) as a storage-proxy configuration. |
fregataa
changed the title
fregataa
changed the title
fregataa
added
urgency:4
…gtask' into feature/improve-vf-purge-using-bgtask
rapsealk
reviewed
Jul 6, 2024
|
|
||
| async with aiotools.TaskGroup() as tg: | ||
| for _, vfolder_iterator in itertools.groupby( | ||
| sorted(requested_vfolders, key=_keyfunc), key=_keyfunc |
There was a problem hiding this comment.
In this case, using typing.Iterable seems more suitable than typing.Sequence for requested_vfolders.
(ref: https://docs.python.org/3/library/functions.html#sorted)
15 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #1797
blocked by #1892we do not need to back-port Graphene relay node part to 23.09.We are now able to backport relay node impl to 23.09Let's not backport this PR
log_levelfieldasyncio.Semaphoreto limit too many concurrent deletion task for one storage-proxy process and apply File-lock to limit concurrent deletion on the same directoryAddmove_to_trash_vfolder,restore_vfolder,delete_forever_vfolderandpurge_vfolderGraphQL endpointsAdding GQL APIs will be handled in another PR
Checklist: (if applicable)
ai.backend.test📚 Documentation preview 📚: https://sorna--1884.org.readthedocs.build/en/1884/
📚 Documentation preview 📚: https://sorna-ko--1884.org.readthedocs.build/ko/1884/