Uploading to cloud platforms: GCP

[dustymabe]

This is part of #146 and tracks the work/discussion around uploading to GCP.

[bgilbert]

I think this involves the following:

• Have GCP create a project to host the images. That project is traditionally called <something>-cloud. We should decide whether the project should be FCOS-specific or Fedora-wide. Our GCP contacts would like to meet with someone from releng to start setting this up.
• Adapt plume for FCOS.
• After testing, have GCP mark the project public.

[dustymabe]

related: coreos/coreos-assembler#493

[bgilbert]

Infra ticket to create the requisite GCP projects.

[cgwalters]

I'm working on some GCP bits for RHCOS (see e.g. openshift/installer#2921 ) and not having FCOS there inhibits doing things upstream first.

[dghubble]

With manually uploaded GCP images, zincati checks for updates and reboots promptly (speedy!) after boot which has some interesting cascading affect on interrupting inital cluster bootstrapping that I noticed today. Being able to use a latest GCP channel image would have a nice side-benefit of (mostly) clearing up immediate reboot issues.

[cgwalters]

@dghubble slightly related coreos/zincati#251
But you probably need to be using https://github.com/coreos/airlock or an equivalent if you aren't.

[LorbusChris]

I think also related here is #392

In OKD we're working around this by adding a etc/zincati/config.d/90-disable-feature.toml that explicitly disables Zincati updates with Ignition: https://github.com/openshift/installer/blob/fcos/data/data/bootstrap/files/etc/zincati/config.d/90-disable-feature.toml

[bgilbert]

@LorbusChris That approach makes sense when the cluster has its own mechanism for updating nodes. In general, though, we shouldn't encourage users to disable updates.

[LorbusChris]

Indeed, we shouldn't encourage that in general. My thinking was that in this specific case of a (probably short-lived) bootstrap host it may make some sense :)

[dghubble]

Thanks for the suggestions. For now I'm content to follow this issue for GCP uploads. Manually uploading new images or sometimes having to retry the bootstrap in this one case is ok to me as I'd like to keep auto-updates enabled (its a master/controller, rather than a toss-away like OKD uses I think).

Mainly just highlighting that stale manual images can cause these reboots (e.g. new instance creation, after GCE preemption) that wouldn't otherwise be needed, until the channel is in place.

[lucab]

@dghubble good feedback points on poseidon/typhoon#687. I have a few more things for you:

1. Zincati only runs after boot-complete.target. If your bootstrapping flow needs to happen before any update, you can delay reaching that target until done.
2. if you are bootstrapping from scratch and can't afford some lock-managing service around, it sounds like you may want something like updates: new strategy based on local filesystem zincati#245 implemented
3. similar to the above, path-activation may be interesting too but I'm unsure on how to plug the default-on and Ignition disabling into that
4. in general some deterministic trigger is preferred over an arbitrary timer/delay. Both the timer-activation and the SSH-delay would just make races harder to detect.

[dghubble]

Thanks @lucab! I think I'd wanna aim for the lock managing approach personally - a local file strategy could help in a pinch here (for bootstrap) and a later an airlock atop the cluster could get (roughly) one at a time node updates (I don't have strong consistency needs). I looked at airlock's etcd strategy, but I'm reluctant to give it access policy-wise or deal with giving it a TLS client cert, etc. I'd picture airlock (or similar) maybe running as a pod with rbac to write a configmap or annotation, some bit of scratch space for locked yes/no. airlock is looking promising and simpler than cluo was, avoiding the coordinator/daemonset.

For 1, I'd worry about blocking the target on future reboots (bootstrap is once per cluster lifetime). 3 yeah is kinda , and agreed that 4 may not be the best for this situation.

Maybe I'll move this to a (new?) issue so I don't hijack.

[dustymabe]

Some updates...

I think this involves the following:

• Have GCP create a project to host the images. That project is traditionally called <something>-cloud. We should decide whether the project should be FCOS-specific or Fedora-wide. Our GCP contacts would like to meet with someone from releng to start setting this up.

We have a fedora-coreos-cloud project now.

• Adapt plume for FCOS.

We adapted ore within COSA to support uploading to GCP and doing what we need.

• mantle/api: gcloud: remove FCOS specific image GuestOSFeatures coreos-assembler#1333
• mantle/ore: glcoud: add ability to specify image description for gcp uploads coreos-assembler#1328
• mantle/ore: gcloud: clean up upload.go, stop mutating image name coreos-assembler#1322
• mantle/ore: gcp: add image family support, add deprecate image functionality coreos-assembler#1319

We modified the pipeline to start doing the upload:

• Add GCP image uploads to the pipeline fedora-coreos-pipeline#219
• Jenkinsfile: add description for the GCP uploaded image fedora-coreos-pipeline#220

• After testing, have GCP mark the project public.

GCP is working on making the fedora-coreos-cloud project public. In the meantime I've individually marked the images from our latest testing and stable releases as public so you should be able to use them today:

# gcloud compute images list --project fedora-coreos-cloud --no-standard-images --show-deprecated 
NAME                                       PROJECT              FAMILY                       DEPRECATED  STATUS
fedora-coreos-31-20200323-3-2-gcp-x86-64   fedora-coreos-cloud  fedora-coreos-stable                     READY
fedora-coreos-31-20200407-2-2-gcp-x86-64   fedora-coreos-cloud  fedora-coreos-testing                    READY

[vrutkovs]

Is there a bucket with .tar.gz artifact uploaded so that OKD could reuse it?

[dustymabe]

the url for the tar.gz within GCP is in the meta.json. My understanding is that it's accessible to any authenticated users. Previously I had considered the file in the image bucket to be purely ephemeral (i.e. only needing to exist for the image import to happen), but I've recently learned of openshift creating images on the fly during install. I'd like to understand why we need to do that if an image is available publicly (and globally) already? Different guestOSfeatures? Encryption?

[cgwalters]

but I've recently learned of openshift creating images on the fly during install. I'd like to understand why we need to do that if an image is available publicly (and globally) already? Different guestOSfeatures? Encryption?

The idea broadly speaking was that RHEL CoreOS is an implementation detail of OpenShift. Hence, there wasn't a desire to publish pre-built images globally. This might change at some point.

It would seem reasonable to me though to change openshift-install to support directly using a public GCP image if it's available, it'd just require some conditionals in the logic for terraform.

[dustymabe]

but I've recently learned of openshift creating images on the fly during install. I'd like to understand why we need to do that if an image is available publicly (and globally) already? Different guestOSfeatures? Encryption?

The idea broadly speaking was that RHEL CoreOS is an implementation detail of OpenShift. Hence, there wasn't a desire to publish pre-built images globally. This might change at some point.

Yeah I think that makes sense. Since we are publishing images already for Fedora CoreOS I think it makes sense to do what you suggest below 👇🏼 for Fedora CoreOS/OKD.

It would seem reasonable to me though to change openshift-install to support directly using a public GCP image if it's available, it'd just require some conditionals in the logic for terraform.

[dustymabe]

remaining items here:

• update mantle to use new image update API so we can create images not in an image family and then later add them to an image family
  • Step 1: PR in gcp: allow for deprecating and then attaching to image family coreos-assembler#1444
  • Step 2: PR in update for GCP image build/release workflow fedora-coreos-pipeline#237
• attach "licenses" to images when they get created (this allows for viewing how many instances are created on each stream)
  • Step 1: PR in mantle/ore: gcloud: add ability to attach license to image coreos-assembler#1446
  • Step 2: PR in update for GCP image build/release workflow fedora-coreos-pipeline#237
• documentation page for GCP GCP documentation page in Fedora CoreOS docs #475
• change image size to be 10GiB or greater (known to have suboptimal I/O performance with <10G)
  • broke out this change into GCP: increase image size to 10G  #492
• run kola CI tests against GCP images
  • PR in Add pipeline to test GCP images fedora-coreos-pipeline#233
  • Debugging upgrade test issues in kola tests fail in GCP environment when running in pipeline #487

[dustymabe]

updated #147 (comment) with current status.

[lucab]

@dghubble yes, feel free to move to another ticket or a forum discussion. From your last reply, it looks like a volatile fragment in /run to use coreos/zincati#245 on first-boot only could be a good solution.

[dustymabe]

updated #147 (comment) with current status.

[dghubble]

I'm using the new uploaded GCP images instead of manually uploading. Many thanks for these!

# Terraform example
# Fedora CoreOS most recent image from stream
data "google_compute_image" "fedora-coreos" {
  project = "fedora-coreos-cloud"
  family  = "fedora-coreos-${var.os_stream}"
}

[dustymabe]

ok all things in #147 (comment) are now done or broken out into other tickets. I also created a new ticket for us getting this plumbed through to our download page: #494

I'm going to mark this as done. It was a long journey.