Implement custom allocation for SWAN event participants with a GPU #216
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PMax5
previously approved these changes
Apr 22, 2024
etejedor
force-pushed
the
training-events
branch
from
048d96a to
1cbecfb
Compare
Context: SWAN hosts events (i.e. trainings) that often require the access to GPUs from participants. Extra resources are provisioned to be able to support such events. The functionality implemented by this commit allows to reserve some GPU resources for exclusive use of the participants of an event. Only pods from those participants (who must belong to an egroup) will be allocated on the event resources. This is useful to guarantee that the participants will have the resources that were agreed with the organisers. Furthermore, if the GPU resources are fragments of MIG GPUs, now we can configure that event pods must request the desired type of fragment, so the matching is properly done at the GPU resource level too. Two configurable parameters are added here for that purpose: - events.role: name of the auth role that participants of a SWAN event have. - events.gpu_name: name of the GPU resource assigned to those participants.
etejedor
force-pushed
the
training-events
branch
from
1cbecfb to
15235ea
Compare
To have all GPU-related logic together in the modify pod hook for computing resources.
etejedor
force-pushed
the
training-events
branch
from
3802cb3 to
231a16a
Compare
|
Implemented all comments from @diocas . |
diocas
reviewed
Apr 24, 2024
| """ | ||
| return True if the user has requested a GPU | ||
| """ | ||
| return "cu" in self.spawner.user_options[self.spawner.lcg_rel_field] |
There was a problem hiding this comment.
Depending on how Rodrigo implements the custom env in the spawner, this might crash ^
There was a problem hiding this comment.
If so, he'll fix it :) This is purely a code move.
| @@ -393,4 +479,11 @@ def computing_modify_pod_hook(spawner, pod): | |||
| return computing_pod_hook_handler.get_swan_user_pod() | |||
|
|
|||
|
|
|||
| # Custom configuration options | |||
| # Name of the role that is assigned to participants of events hosted by SWAN | |||
| events_role = get_config('custom.events.role', 'swan-events') | |||
There was a problem hiding this comment.
Why did you declare the vars here instead of inside the method that uses it? It would be cleaner if in the future we remove that method.
There was a problem hiding this comment.
I followed the same structure as in the other hooks that have custom config. E.g. see https://github.com/swan-cern/swan-charts/blob/master/swan-cern/files/swan_config_cern.py#L232-L234
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context: SWAN hosts events (i.e. trainings) that often require the access to GPUs from participants. Extra resources are provisioned to be able to support such events.
The functionality implemented by this commit allows to reserve some GPU resources for exclusive use of the participants of an event. Only pods from those participants (who must belong to an egroup) will be allocated on the event resources. This is useful to guarantee that the participants will have the resources that were agreed with the organisers.
Furthermore, if the GPU resources are fragments of MIG GPUs, now we can configure that event pods must request the desired type of fragment, so the matching is properly done at the GPU resource level too.