Deploy gcp-filestore-backups for GCP shared cluster #4447

sgibson91 · 2024-07-18T12:00:39Z

fixes Deploy gcp-filestore-backup to the 2i2c shared cluster on GCP #4397

enables relevant resources in terraform
enables gcp-filestore-backups in support chart for 2i2c cluster
makes chartpress able to write the image name and tag into the support chart values
applies any fixes required to allow code to function

sgibson91 · 2024-07-18T12:01:09Z

Output of tf plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # google_container_cluster.cluster will be updated in-place
  ~ resource "google_container_cluster" "cluster" {
        id                                       = "projects/two-eye-two-see/locations/us-central1-b/clusters/pilot-hubs-cluster"
        name                                     = "pilot-hubs-cluster"
        # (28 unchanged attributes hidden)

        # (21 unchanged blocks hidden)
    }

  # google_monitoring_notification_channel.pagerduty_disk_space will be updated in-place
  ~ resource "google_monitoring_notification_channel" "pagerduty_disk_space" {
        id           = "projects/two-eye-two-see/notificationChannels/2203982305292521130"
        name         = "projects/two-eye-two-see/notificationChannels/2203982305292521130"
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

  # google_project_iam_custom_role.filestore_backups[0] will be created
  + resource "google_project_iam_custom_role" "filestore_backups" {
      + deleted     = (known after apply)
      + description = "Minimal role for gcp-filestore-backups pods on pilot-hubs to identify as current project"
      + id          = (known after apply)
      + name        = (known after apply)
      + permissions = [
          + "file.backups.*",
        ]
      + project     = "two-eye-two-see"
      + role_id     = "pilot_hubs_filestore_backups"
      + stage       = "GA"
      + title       = "Identify as project role for pods in pilot-hubs"
    }

  # google_project_iam_member.filestore_backups_binding[0] will be created
  + resource "google_project_iam_member" "filestore_backups_binding" {
      + etag    = (known after apply)
      + id      = (known after apply)
      + member  = (known after apply)
      + project = "two-eye-two-see"
      + role    = (known after apply)
    }

  # google_service_account.filestore_backup_sa[0] will be created
  + resource "google_service_account" "filestore_backup_sa" {
      + account_id   = "pilot-hubs-filestore-backup"
      + disabled     = false
      + display_name = "Service account for gcp-filestore-backups pods in pilot-hubs"
      + email        = (known after apply)
      + id           = (known after apply)
      + member       = (known after apply)
      + name         = (known after apply)
      + project      = "two-eye-two-see"
      + unique_id    = (known after apply)
    }

  # google_service_account_iam_binding.filestore_backups_binding[0] will be created
  + resource "google_service_account_iam_binding" "filestore_backups_binding" {
      + etag               = (known after apply)
      + id                 = (known after apply)
      + members            = [
          + "serviceAccount:two-eye-two-see.svc.id.goog[support/gcp-filestore-backups-sa]",
        ]
      + role               = "roles/iam.workloadIdentityUser"
      + service_account_id = (known after apply)
    }

  # google_service_account_key.cd_sa will be updated in-place
  ~ resource "google_service_account_key" "cd_sa" {
        id                 = "projects/two-eye-two-see/serviceAccounts/[email protected]/keys/ee245c75f6c3fa5a36ae59597b356f4c6da80334"
        name               = "projects/two-eye-two-see/serviceAccounts/[email protected]/keys/ee245c75f6c3fa5a36ae59597b356f4c6da80334"
        # (8 unchanged attributes hidden)
    }

  # google_service_account_key.registry_sa_keys["binder-staging"] will be updated in-place
  ~ resource "google_service_account_key" "registry_sa_keys" {
        id                 = "projects/two-eye-two-see/serviceAccounts/binder-staging-registry-sa@two-eye-two-see.iam.gserviceaccount.com/keys/195042aa9b5edc2b3502736bd2d99daa7c5c9877"
        name               = "projects/two-eye-two-see/serviceAccounts/binder-staging-registry-sa@two-eye-two-see.iam.gserviceaccount.com/keys/195042aa9b5edc2b3502736bd2d99daa7c5c9877"
        # (8 unchanged attributes hidden)
    }

  # google_service_account_key.registry_sa_keys["binderhub-ui-demo"] will be updated in-place
  ~ resource "google_service_account_key" "registry_sa_keys" {
        id                 = "projects/two-eye-two-see/serviceAccounts/binderhub-ui-demo-registry-sa@two-eye-two-see.iam.gserviceaccount.com/keys/7ad6294d83c8adb31c2e286e56f61a7d9f447f13"
        name               = "projects/two-eye-two-see/serviceAccounts/binderhub-ui-demo-registry-sa@two-eye-two-see.iam.gserviceaccount.com/keys/7ad6294d83c8adb31c2e286e56f61a7d9f447f13"
        # (8 unchanged attributes hidden)
    }

Plan: 4 to add, 5 to change, 0 to destroy.

Changes to Outputs:
  + gcp_filestore_backups_k8s_sa_annotations = (known after apply)
  ~ regular_channel_latest_k8s_versions      = {
      ~ "1."    = "1.29.5-gke.1091002" -> "1.29.6-gke.1038001"
      ~ "1.27." = "1.27.14-gke.1042001" -> "1.27.14-gke.1059002"
      ~ "1.28." = "1.28.10-gke.1075001" -> "1.28.11-gke.1019001"
      ~ "1.29." = "1.29.5-gke.1091002" -> "1.29.6-gke.1038001"
    }

github-actions · 2024-07-18T12:22:53Z

Merging this PR will trigger the following deployment actions.

Support and Staging deployments

Cloud Provider	Cluster Name	Upgrade Support?	Reason for Support Redeploy	Upgrade Staging?	Reason for Staging Redeploy
aws	nasa-veda	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	leap	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	pangeo-hubs	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	jupyter-health	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	linked-earth	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	nasa-ghg	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	gridsst	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	cloudbank	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	nasa-cryo	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	catalystproject-africa	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	smithsonian	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
kubeconfig	pchub	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	openscapes	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	opensci	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	nasa-esdis	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	2i2c-aws-us	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	2i2c-uk	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	kitware	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	awi-ciroh-2	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	catalystproject-latam	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
kubeconfig	utoronto	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	2i2c	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	jupyter-meets-the-earth	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	victor	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	ubc-eoas	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	earthscope	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
gcp	hhmi	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified
aws	projectpythia	Yes	Support helm chart has been modified	Yes	Core infrastructure has been modified

Production deployments

Cloud Provider	Cluster Name	Hub Name	Reason for Redeploy
aws	nasa-veda	prod	Core infrastructure has been modified
gcp	leap	prod	Core infrastructure has been modified
gcp	leap	public	Core infrastructure has been modified
gcp	pangeo-hubs	prod	Core infrastructure has been modified
gcp	pangeo-hubs	coessing	Core infrastructure has been modified
aws	jupyter-health	prod	Core infrastructure has been modified
gcp	linked-earth	prod	Core infrastructure has been modified
aws	nasa-ghg	prod	Core infrastructure has been modified
aws	gridsst	prod	Core infrastructure has been modified
gcp	cloudbank	bcc	Core infrastructure has been modified
gcp	cloudbank	ccsf	Core infrastructure has been modified
gcp	cloudbank	csm	Core infrastructure has been modified
gcp	cloudbank	dvc	Core infrastructure has been modified
gcp	cloudbank	elcamino	Core infrastructure has been modified
gcp	cloudbank	evc	Core infrastructure has been modified
gcp	cloudbank	glendale	Core infrastructure has been modified
gcp	cloudbank	high	Core infrastructure has been modified
gcp	cloudbank	howard	Core infrastructure has been modified
gcp	cloudbank	miracosta	Core infrastructure has been modified
gcp	cloudbank	riohondo	Core infrastructure has been modified
gcp	cloudbank	skyline	Core infrastructure has been modified
gcp	cloudbank	demo	Core infrastructure has been modified
gcp	cloudbank	fresno	Core infrastructure has been modified
gcp	cloudbank	humboldt	Core infrastructure has been modified
gcp	cloudbank	laney	Core infrastructure has been modified
gcp	cloudbank	sbcc	Core infrastructure has been modified
gcp	cloudbank	sbcc-dev	Core infrastructure has been modified
gcp	cloudbank	elac	Core infrastructure has been modified
gcp	cloudbank	lacc	Core infrastructure has been modified
gcp	cloudbank	lamission	Core infrastructure has been modified
gcp	cloudbank	lbcc	Core infrastructure has been modified
gcp	cloudbank	mendocino	Core infrastructure has been modified
gcp	cloudbank	mills	Core infrastructure has been modified
gcp	cloudbank	mission	Core infrastructure has been modified
gcp	cloudbank	moreno	Core infrastructure has been modified
gcp	cloudbank	norco	Core infrastructure has been modified
gcp	cloudbank	palomar	Core infrastructure has been modified
gcp	cloudbank	pasadena	Core infrastructure has been modified
gcp	cloudbank	reedley	Core infrastructure has been modified
gcp	cloudbank	sjcc	Core infrastructure has been modified
gcp	cloudbank	sacramento	Core infrastructure has been modified
gcp	cloudbank	srjc	Core infrastructure has been modified
gcp	cloudbank	saddleback	Core infrastructure has been modified
gcp	cloudbank	santiago	Core infrastructure has been modified
gcp	cloudbank	sjsu	Core infrastructure has been modified
gcp	cloudbank	sierra	Core infrastructure has been modified
gcp	cloudbank	tuskegee	Core infrastructure has been modified
gcp	cloudbank	wlac	Core infrastructure has been modified
gcp	cloudbank	csulb	Core infrastructure has been modified
gcp	cloudbank	csum	Core infrastructure has been modified
aws	nasa-cryo	prod	Core infrastructure has been modified
aws	catalystproject-africa	nm-aist	Core infrastructure has been modified
aws	catalystproject-africa	must	Core infrastructure has been modified
aws	catalystproject-africa	uvri	Core infrastructure has been modified
aws	catalystproject-africa	wits	Core infrastructure has been modified
aws	catalystproject-africa	kush	Core infrastructure has been modified
aws	catalystproject-africa	molerhealth	Core infrastructure has been modified
aws	catalystproject-africa	aibst	Core infrastructure has been modified
aws	catalystproject-africa	bhki	Core infrastructure has been modified
aws	catalystproject-africa	bon	Core infrastructure has been modified
aws	smithsonian	prod	Core infrastructure has been modified
kubeconfig	pchub	prod	Core infrastructure has been modified
aws	openscapes	prod	Core infrastructure has been modified
aws	openscapes	workshop	Core infrastructure has been modified
aws	opensci	sciencecore	Core infrastructure has been modified
aws	opensci	climaterisk	Core infrastructure has been modified
aws	opensci	small-binder	Core infrastructure has been modified
aws	opensci	big-binder	Core infrastructure has been modified
aws	nasa-esdis	prod	Core infrastructure has been modified
aws	2i2c-aws-us	showcase	Core infrastructure has been modified
aws	2i2c-aws-us	ncar-cisl	Core infrastructure has been modified
aws	2i2c-aws-us	itcoocean	Core infrastructure has been modified
aws	2i2c-aws-us	cosmicds	Core infrastructure has been modified
aws	2i2c-aws-us	isea	Core infrastructure has been modified
aws	2i2c-aws-us	neurohackademy	Core infrastructure has been modified
gcp	2i2c-uk	lis	Core infrastructure has been modified
aws	kitware	prod	Core infrastructure has been modified
gcp	awi-ciroh-2	prod	Core infrastructure has been modified
gcp	catalystproject-latam	unitefa-conicet	Core infrastructure has been modified
gcp	catalystproject-latam	cicada	Core infrastructure has been modified
gcp	catalystproject-latam	gita	Core infrastructure has been modified
gcp	catalystproject-latam	iner	Core infrastructure has been modified
gcp	catalystproject-latam	plnc	Core infrastructure has been modified
gcp	catalystproject-latam	unam	Core infrastructure has been modified
gcp	catalystproject-latam	uprrp	Core infrastructure has been modified
gcp	catalystproject-latam	cabana	Core infrastructure has been modified
gcp	catalystproject-latam	nnb-ccg	Core infrastructure has been modified
gcp	catalystproject-latam	labi	Core infrastructure has been modified
kubeconfig	utoronto	prod	Core infrastructure has been modified
kubeconfig	utoronto	r-prod	Core infrastructure has been modified
gcp	2i2c	imagebuilding-demo	Core infrastructure has been modified
gcp	2i2c	binderhub-ui-demo	Core infrastructure has been modified
gcp	2i2c	demo	Core infrastructure has been modified
gcp	2i2c	ohw	Core infrastructure has been modified
gcp	2i2c	aup	Core infrastructure has been modified
gcp	2i2c	temple	Core infrastructure has been modified
gcp	2i2c	ucmerced	Core infrastructure has been modified
gcp	2i2c	climatematch	Core infrastructure has been modified
gcp	2i2c	mtu	Core infrastructure has been modified
gcp	2i2c	tufts	Core infrastructure has been modified
aws	jupyter-meets-the-earth	prod	Core infrastructure has been modified
aws	victor	prod	Core infrastructure has been modified
aws	ubc-eoas	prod	Core infrastructure has been modified
aws	earthscope	prod	Core infrastructure has been modified
gcp	hhmi	prod	Core infrastructure has been modified
gcp	hhmi	spyglass	Core infrastructure has been modified
gcp	hhmi	binder	Core infrastructure has been modified
aws	projectpythia	prod	Core infrastructure has been modified
aws	projectpythia	pythia-binder	Core infrastructure has been modified

sgibson91 · 2024-07-18T12:30:32Z

First crash: gcloud needs a read-writeable filesystem

WARNING: Could not setup log file in /.config/gcloud/logs, (OSError: [Errno 30] Read-only file system: '/.config'.
The configuration directory may not be writable. To learn more, see https://cloud.google.com/sdk/docs/configurations#creating_a_configuration
ERROR: gcloud crashed (OSError): [Errno 30] Read-only file system: '/.config'

If you would like to report this issue, please run the following command:
  gcloud feedback

To check gcloud for common problems, please run the following command:
  gcloud info --run-diagnostics
Traceback (most recent call last):
  File "//gcp-filestore-backups.py", line 248, in <module>
    main(args)
  File "//gcp-filestore-backups.py", line 195, in main
    filestore_backups = get_existing_backups(
                        ^^^^^^^^^^^^^^^^^^^^^
  File "//gcp-filestore-backups.py", line 38, in get_existing_backups
    backups = subprocess.check_output(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['gcloud', 'filestore', 'backups', 'list', '--format=json', '--project=two-eye-two-see', '--region=us-central1']' returned non-zero exit status 1.

…e that subfolder the working dir

…e tag in support values

… user 1000

sgibson91 · 2024-07-18T15:22:34Z

@yuvipanda this is now working and has created a backup https://console.cloud.google.com/filestore/instances/locations/us-central1-b/id/pilot-hubs-homedirs;tab=backups?project=two-eye-two-see

but I did have to set readOnlyRootFilesystem: False so that gcloud was able to write it's log files - is that ok or is there a better practice to solve this problem?

yuvipanda · 2024-07-18T16:17:04Z

AMAZING JOB @sgibson91!

re: the readonly filesystem, I think in more security sensitive cases we may try to get gcloud to write its config or log files to a different place. However, in this case, I think it's fine. I'd recommend writing a comment about why that setting is False and then calling it done.

sgibson91 · 2024-07-18T16:40:23Z

we may try to get gcloud to write its config or log files to a different place.

Yeah, that is what I was trying to do with the additional lines in the dockerfile, but just didn't work until I flipped that boolean

github-actions · 2024-07-19T10:51:28Z

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/10006870823

enable filestore backups for shared gcp cluster in terraform

e6ca360

github-actions bot assigned sgibson91 Jul 18, 2024

sgibson91 added 3 commits July 18, 2024 13:05

Fix permission definition for filestore backups iam custom role

0b4d000

Enable gcpFilestoreBackups in 2i2c cluster support chart values

4c38334

Explicitly set pod .spec.template.spec.strategy.type to a string

c5e1393

Put deployment strategy key in the right place

7aceddd

sgibson91 added 7 commits July 18, 2024 13:56

Move gcp-filestore-backups script into subfolder in container and mak…

eb46be9

…e that subfolder the working dir

Update image tag for gcp-filestore-backups image

45554cf

Make it possible for chartpress to set the gcp-filestore-backups imag…

6f760b3

…e tag in support values

Set env var for gcloud config location

1861bf0

Update image tag

d8dceae

Make an /app/.config/gcloud folder for gclkoud to use and chown it to…

f5351ee

… user 1000

Bump image tag again

601cd65

sgibson91 force-pushed the deploy-gcp-filestore-backups branch from d0f58f2 to 601cd65 Compare July 18, 2024 15:13

Set readOnlyRootFilesystem to False

5c42ca0

sgibson91 marked this pull request as ready for review July 18, 2024 15:22

Add a comment explaining why we set readOnlyFilesystem to False

e9a1d20

sgibson91 merged commit 8d3db42 into 2i2c-org:main Jul 19, 2024
36 checks passed

sgibson91 deleted the deploy-gcp-filestore-backups branch July 19, 2024 10:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deploy gcp-filestore-backups for GCP shared cluster #4447

Deploy gcp-filestore-backups for GCP shared cluster #4447

sgibson91 commented Jul 18, 2024 •

edited

Loading

sgibson91 commented Jul 18, 2024

github-actions bot commented Jul 18, 2024 •

edited

Loading

sgibson91 commented Jul 18, 2024

sgibson91 commented Jul 18, 2024 •

edited

Loading

yuvipanda commented Jul 18, 2024

sgibson91 commented Jul 18, 2024

github-actions bot commented Jul 19, 2024

Deploy gcp-filestore-backups for GCP shared cluster #4447

Deploy gcp-filestore-backups for GCP shared cluster #4447

Conversation

sgibson91 commented Jul 18, 2024 • edited Loading

sgibson91 commented Jul 18, 2024

github-actions bot commented Jul 18, 2024 • edited Loading

Support and Staging deployments

Production deployments

sgibson91 commented Jul 18, 2024

sgibson91 commented Jul 18, 2024 • edited Loading

yuvipanda commented Jul 18, 2024

sgibson91 commented Jul 18, 2024

github-actions bot commented Jul 19, 2024

sgibson91 commented Jul 18, 2024 •

edited

Loading

github-actions bot commented Jul 18, 2024 •

edited

Loading

sgibson91 commented Jul 18, 2024 •

edited

Loading