klib(cloud_init): pre_download functionality - fetch and replace

[rinor]

INTRO

cloud_init klib, among other functionalities, upon instance start, has the ability to:

1. download files from a static and predefined url(upon initial image creation), and store them in the provided destination (depending on the config), i.e: one might configure smth like this so the instance(s) will download the program binary upon every start, useful in different scenarios (dev and/or prod) to distribute a program/config/file upgrade/update without rebuilding the image and redeploying new instance.

{
  "ManifestPassthrough": {
    "cloud_init": {
      "download": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/serviceA/serviceA_binary_global",
          "dest": "/program",
          "overwrite": "t"
        }
      ]
    }
  }
}

2. get a json output from a static and predefined url(upon initial image creation), and set environment variables that can be used later on by the program.

{
  "ManifestPassthrough": {
    "cloud_init": {
      "download_env": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/serviceA/serviceA-01_env.json"
        }
      ]
    }
  }
}

Now, if you want to deploy another instance of the same service, but with configuration (as an example) that differs from serviceA-01_env.json, the rationale is to create a new image with updated configuration (serviceA-02_env.json) and deploy new instance based of that image.
So cloud_init config might look like

{
  "ManifestPassthrough": {
    "cloud_init": {
      "download_env": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/serviceA/serviceA-02_env.json"
        }
      ]
    }
  }
}

if you need another one, you repeat the process serviceA-03_env.json -> image -> instance, and that's fine and manageable, but becomes a bit annoying when the number of instances increases, since for each instance you need a dedicated image (built and upload) .

So when/if the time comes to upgrade Nanos itself (bug fix, improvements, ...) you have to repeat the image build -> image upload process for every instance you want to update. Again, this is doable and manageable most of the times.

In this simplified case scenario, the only reason we need a dedicated image for each instance, is related to the static cloud_init configuration.

Yes, under certain conditions, there are workarounds to use a single image and provide dynamic data from the same url that one can implement on the provisioner (things like: detect client ip on the request, find instance id/name/hostname and that might be enough for you, if not query metadata server and get some value(s) (that can be updated) to help you decide and so on...). You could even use a provisioning binary, downloaded from cloud_init that could take care of downloading the "real" program and/or file configs from the provisioner server, since it can query the internal metadata server it has a lot of information to use, but imho it's more annoying/complicated than it needs to be.

---

Proposal

what if we could extend cloud_init with some functionality that can help and provide some dynamic values that can be used on download and download_env attributes (initially).

So the general naive idea is smth like this, where in this case we query the gcp metadata server for custom metadata items and store the output value to a dst key (basically a kv) and then use that to dynamically replace the key pattern on the download and download_env src url(s) at least.

It might looks similar to this, but needs to be discussed and agreed upon:

{
  "ManifestPassthrough": {
    "cloud_init": {
      "pre_download": [
        {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/service",
          "header": "Metadata-Flavor: Google",
          "dst": "#_SERVICE_#"
        },
        {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/serviceenv",
          "header": "Metadata-Flavor: Google",
          "dst": "#_ENVFILE_#"
        },
        {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/serviceprogram",
          "header": "Metadata-Flavor: Google",
          "dst": "#_PROGRAM_#"
        }
      ]
      "download_env": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/#_SERVICE_#/#_ENVFILE_#"
        }
      ],
      "download": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/#_SERVICE_#/#_PROGRAM_#",
          "dest": "/program",
          "overwrite": "t"
        }
      ]
    }
  }
}

[francescolavra]

That makes sense to me. The klib would have to parse the "src" parameters from the "download" and "download_env" configuration, and if there is one or more tokens identified by the #_ and _# delimiters, look for a corresponding "pre_download" entry and retrieve the data from the relevant url. This could be used not only for the "src" parameter but also for "auth", so the klib could retrieve e.g. an authentication token from the metadata server and use it for downloading files and environment variables.
Maybe I would not use the #_ and _# delimiters in each "pre_download" key, just the key name (e.g. "SERVICE", "ENVFILE", etc.) would be sufficient. And since there can only be one "pre_download" entry per key, perhaps I would use JSON attribute names for the keys instead of having a "dst" attribute, e.g. something like below:

      "pre_download": [
        "SERVICE": {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/service",
          "header": "Metadata-Flavor: Google"
        },
        "ENVFILE": {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/serviceenv",
          "header": "Metadata-Flavor: Google,
        },
        "PROGRAM": {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/serviceprogram",
          "header": "Metadata-Flavor: Google"
        }
      ]

[rinor]

So what would be the preferred (easiest/fastest to parse on nanos) config format?

• 1 - smth like this - (I like/prefer this one)

{
  "ManifestPassthrough": {
    "cloud_init": {
      "pre_download": {
        "SERVICE": {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/service",
          "header": "Metadata-Flavor: Google"
        },
        "ENVFILE": {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/serviceenv",
          "header": "Metadata-Flavor: Google"
        },
        "PROGRAM": {
          "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/serviceprogram",
          "header": "Metadata-Flavor: Google"
        }
      },
      "download_env": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/SERVICE/ENVFILE"
        }
      ],
      "download": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/SERVICE/PROGRAM",
          "dest": "/program",
          "overwrite": "t"
        }
      ]
    }
  }
}

• 2 - or like this

{
  "ManifestPassthrough": {
    "cloud_init": {
      "pre_download": [
        {
          "SERVICE": {
            "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/service",
            "header": "Metadata-Flavor: Google"
          }
        },
        {
          "ENVFILE": {
            "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/serviceenv",
            "header": "Metadata-Flavor: Google"
          }
        },
        {
          "PROGRAM": {
            "src": "http://169.254.169.254/computeMetadata/v1/instance/attributes/serviceprogram",
            "header": "Metadata-Flavor: Google"
          }
        }
      ],
      "download_env": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/SERVICE/ENVFILE"
        }
      ],
      "download": [
        {
          "auth": "Basic xxxxxxxxxxxxx",
          "src": "http://192.168.0.10:8080/prov/SERVICE/PROGRAM",
          "dest": "/program",
          "overwrite": "t"
        }
      ]
    }
  }
}

[francescolavra]

I also prefer config format 1, which is easier to parse

[eyberg]

some of this is ok but I'm not comfortable w/enabling a binary distribution like this; nanos instances aren't meant to be "upgraded" - they're meant to be replaced w/upgrades and the fewer loopholes that people have to inject things the better

[eyberg]

I apologize - I missed this issue earlier and it seems important in the fact that I think it exposes a few misconceptions wrt to this ask and some other recent related ones and this might clear some things up:

we actually explicitly disallow one of these use-cases via https://docs.ops.city/ops/configuration#program_overwrite and I would go as far to say that should be something that can only be opened up in a custom nanos debug build (eg: no releases should be able to enable it)

the number of instances increases, since for each instance you need a dedicated image (built and upload) .

This is not true but I think for your use case it might be.

Images are not supposed to be shared amongst different programs. We treat each image as a program. It's one thing if you are spinning up X instances that are all the same image behind a load balancer - in that case you can use the same image but if it's a totally different program then you are going to have a custom image regardless. We have no plans to support having some sort of 'generic' image.

You could even use a provisioning binary, downloaded from cloud_init that could take care of downloading the "real" program

This is another thing that we will not support. We obviously have the elf loader itself but that is about as far as you're going to get here. We definitely would not want to support this use-case either.

[rinor]

There seems to be a misunderstanding, probably I wasn't clear enough, regarding the production program deployment. I'm not advocating about generic images or program upgrades in prod. There the deployments are "locked" and readonly_rootfs in on and any upgrade is to be performed via a new image.

The reason "program" was brought up here, in retrospect I should have used a configuration file instead, was to point out that:

1. we want to perform the replacements in both config items (download_env and download)
2. we are using the same program (not different programs) everywhere, but with different config needs for each instance.

Yes, under certain conditions, there are workarounds to use a single image and provide dynamic data from the same url that one can implement on the provisioner (things like: detect client ip on the request, find instance id/name/hostname and that might be enough for you, if not query metadata server and get some value(s) (that can be updated) to help you decide and so on...). You could even use a provisioning binary, downloaded from cloud_init that could take care of downloading the "real" program and/or file configs from the provisioner server, since it can query the internal metadata server it has a lot of information to use, but imho it's more annoying/complicated than it needs to be.

As for this part, it wasn't a request for features and implementations on nanos, just mentioning workarounds used mostly on dev environments.