jupyter-ui: Please allow either parameterization of jupyter-ui-config-config configmap or option for controlling affinityConfig

[Vultaire]

We have a customer which wants to customize their affinityConfig. It seems like the charm currently creates a configmap from a static file, but doesn't offer parameterization of this configmap.

The customer is manually modifying the configmap to something like this:

      affinityConfig:
        # If readonly, the default value will be the only option
        # value is a list of `configKey`s that we want to be selected by default
        value: "gpu-product-NVIDIA-A100-80GB-PCIe"
        # The list of available affinity configs
        # options: []
        options:
          - configKey: "gpu-product-NVIDIA-A100-80GB-PCIe"
            displayName: "nvidia.com/gpu-product"
            affinity:
             # (Require) Node having label: `node_pool=notebook-n1-standard-2`
              nodeAffinity:
                requiredDuringSchedulingIgnoredDuringExecution:
                  nodeSelectorTerms:
                    - matchExpressions:
                        - key: "nvidia.com/gpu.product"
                          operator: "In"
                          values:
                           - "NVIDIA-A100-80GB-PCIe"

It would be preferable for this to be something managed by the charm rather than an unmanaged change in the environment. Can we add functionality to allow this type of edit in some way?

[ca-scribner]

The workaround @Vultaire mentions is fragile but works for Charmed Kubeflow 1.6 (fragile because that configmap can be rewritten by juju, for example if the jupyter-ui unit is restarted). For Charmed Kubeflow 1.7 though, that configmap is no longer used and instead we push the configuration straight into the container. The workaround might still work, depending on which config the workload pod looks at first, but we'd have to check. Either way, it is very hacky.

The UI provides no custom way to set this on a pod-by-pod basis, so providing users with configurations must be done through this spawner config file:

As a workaround for 1.7, you might be able to copy the modified spawner_ui_config.yaml into the unit and overwrite the default, that way the charm pushes your modified version to the container. Something like juju scp spawner_ui_config.yaml jupyter-ui/0:/var/lib/juju/agents/unit-jupyter-ui-0/charm/src/spawner_ui_config.yaml. I think you'd need to also trigger execution of the on_pebble_ready for that container to re-push the file.

[kimwnasptd]

Hi @Vultaire, unfortunately this is something that is currently not supported in Charmed KF (neither 1.6 not 1.7). We'll need to properly implement this feature in Charmed Kubeflow in a follow up release.

Ideally we would like to avoid having clients to interact with the whole YAML config file and, as you mentioned, expose the necessary parts in our Charms. But unfortunately this is currently not supported

[kimwnasptd]

I'll also summarize @ca-scribner's comment, since it's a workaround but I need to stress that this should not be considered a stable workaround. First because the file can get overwritten back to its initial state but more importantly because it expects users to mess with the whole config file, which is too complicated.

Charmed KF 1.6

Modify the ConfigMap in the kubeflow namespace. But since JuJu "owns" this ConfigMap it's expected that it gets reverted to its initial state once in a few days. We've seen this happen with #29 (comment)

Charmed KF 1.7

As @ca-scribner mentioned in Charmed KF 1.7 there is no longer such a ConfigMap. This file is injected to the application container via the Charm.

So in order to override it we'd need to manually push the file into the container and overwrite it with

juju scp spawner_ui_config.yaml jupyter-ui/0:/var/lib/juju/agents/unit-jupyter-ui-0/charm/src/spawner_ui_config.yaml

(just copied the command from above. The initial config file can be found here https://github.com/canonical/notebook-operators/blob/track/1.7/charms/jupyter-ui/src/spawner_ui_config.yaml)

but again if for any reason (i.e. JuJu) restarts the container then this file will be reverted to its original state

[amouu]

Charmed KF 1.7
@ca-scribner May I ask how to trigger execution of the on_pebble_ready for that container to re-push the file?

[nishant-dash]

Hey @kimwnasptd
Is there any update on this?

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-5354.

This message was autogenerated

[ca-scribner]

@nishant-dash we're speccing this now and should implement it in the next few weeks. See here for progress

[nsxsnx]

Hello guys,
Any update on this feature, please?
It would be very useful in heterogeneous clusters.

[ca-scribner]

I'm not involved with these charms anymore, but I think this can be closed because of config options like these.

[orfeas-k]

As @ca-scribner mentioned, this now can be closed. For more details, you can view the relevant documentation https://charmed-kubeflow.io/docs/configure-kubeflow-notebook-creation-page#node-affinities.