Kit changes for Nvidia settings APIs

[monirul]

Issue number:

Closes #

Description of changes:
This PR contains the changes required to support Nvidia settings APIs, settings.nvidia-container-runtime and settings.kubernetes.nvidia

Testing done:
Yes.

bash-5.1# apiclient set settings.kubernetes.device-plugins.nvidia.pass-device-specs=true
bash-5.1# cat /etc/nvidia-k8s-device-plugin/settings.yaml
version: v1
flags:
  migStrategy: "none"
  failOnInitError: true
  plugin:
    passDeviceSpecs: true
    deviceListStrategy: volume-mounts
    deviceIDStrategy: index
bash-5.1# apiclient set settings.kubernetes.device-plugins.nvidia.device-list-strategy=envvar
bash-5.1# cat /etc/nvidia-k8s-device-plugin/settings.yaml
version: v1
flags:
  migStrategy: "none"
  failOnInitError: true
  plugin:
    passDeviceSpecs: true
    deviceListStrategy: envvar
    deviceIDStrategy: index
bash-5.1# apiclient set settings.kubernetes.device-plugins.nvidia.device-id-strategy=uuid
bash-5.1# cat /etc/nvidia-k8s-device-plugin/settings.yaml
version: v1
flags:
  migStrategy: "none"
  failOnInitError: true
  plugin:
    passDeviceSpecs: true
    deviceListStrategy: envvar
    deviceIDStrategy: uuid

bash-5.1# cat /etc/nvidia-container-runtime/config.toml
accept-nvidia-visible-devices-as-volume-mounts = true
accept-nvidia-visible-devices-envvar-when-unprivileged = false

[nvidia-container-cli]
root = "/"
path = "/usr/bin/nvidia-container-cli"
environment = []
ldconfig = "@/sbin/ldconfig"
bash-5.1# apiclient set settings.nvidia-container-runtime.visible-devices-as-volume-mounts=false
bash-5.1# apiclient set settings.nvidia-container-runtime.visible-devices-envvar-when-unprivileged=true
bash-5.1# cat /etc/nvidia-container-runtime/config.toml
accept-nvidia-visible-devices-as-volume-mounts = false
accept-nvidia-visible-devices-envvar-when-unprivileged = true

[nvidia-container-cli]
root = "/"
path = "/usr/bin/nvidia-container-cli"
environment = []
ldconfig = "@/sbin/ldconfig"

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[cbgbt]

Will this result in e.g. "volume-mounts" being double-quoted if the setting is unset?

[monirul]

yes.

[cbgbt]

You mentioned when chatting that we put a default configuration file in place to keep compatibility with downstream variants that won't add the device plugin settings. That makes sense.

An alternative to overwriting a config tempfile could be to place a systemd dropin that overwrites ExecStart based on settings changes. So this file would remain the same, but a template could be rendered to /etc/sysetmd/system/nvidia-k8s-device-plugin.service.d/exec-start adding the alternative line with a reference to the rendered config file.

I know we do something like this for kubelet, though I'm not sure if there are compelling reasons to prefer one to the other. I suppose anything that gets rendered to /etc consumes system memory, so perhaps it's slightly nicer to avoid doing that if we can.

[monirul]

changed as suggested

[cbgbt]

This shouldn't need to be created by tmpfiles.d because the template rendering system will make the directory on our behalf.

[monirul]

removed the tmpfiles as suggested.

[cbgbt]

Did you test that this works correctly on instances without an nvidia device plugin set?

[monirul]

Yes. I tested it if settings.kubernetes.device-plugins.nvidia is not defined then this file is rendered as

[Service]

which are kind of ignored as there is no overriding settings. It works as expected.

[yeazelm]

This feels weird and to someone just seeing this on the system would appear to be a failure? Can we add a comment in an else statement to say deliberately left this way or something? Is it not better to just remove the [Service] as well?

[cbgbt]

I believe you could similarly drop the tmpfiles requirement here since both files are templates.

[monirul]

deleted as suggested.

[yeazelm]

This feels weird and to someone just seeing this on the system would appear to be a failure? Can we add a comment in an else statement to say deliberately left this way or something? Is it not better to just remove the [Service] as well?