Merge branch 'GoogleCloudPlatform:main' into main

.ci/README.md

@@ -42,18 +42,17 @@ Don't panic - this is all quite safe and we have fixed it before.  We store the
 
 It's possible for a job to be cancelled or fail in the middle of pushing downstreams in a transient way.  The sorts of failures that happen at scale - lightning strikes a datacenter (ours or GitHub's!) or some other unlikely misfortune happens.  This has a chance to cause a hiccup in the downstream history, but isn't dangerous.  If that happens, the sync tags may need to be manually updated to sit at the same commit, just before the commit which needs to be generated, or some failed tasks might need to be run by hand.
 
-Updating the sync tags is done like this:
-First, check their state: `git fetch origin && git rev-parse origin/tpg-sync origin/tpgb-sync origin/tf-oics-sync origin/tgc-sync` will list the commits for each of the sync tags.
-(If you have changed the name of the `googlecloudplatform/magic-modules` remote from `origin`, substitute that name instead)
+First, check their state: `git fetch origin && git rev-parse origin/tpg-sync origin/tpgb-sync origin/tf-oics-sync origin/tgc-sync` will list the commits for each of the sync tags. (If you have changed the name of the `GoogleCloudPlatform/magic-modules` remote from `origin`, substitute that name instead, such as `git fetch upstream && git rev-parse upstream/tpg-sync upstream/tpgb-sync upstream/tf-oics-sync upstream/tgc-sync`)
+
 In normal, steady-state operation, these tags will all be identical.  When a failure occurs, some of them may be one commit ahead of the others.  It is rare for any of them to be 2 or more commits ahead of any other.  If some of them are one commit ahead of the others, and there is no pusher task currently running, this means you need to reset them by hand and rerun the failed jobs.  If they diverge by more than one commit, or a pusher task is currently running, you will need to manually run missing tasks.
 
 ### Divergence by zero commits
 
-Just click retry on the failed job in Cloud Build.  Yay!
+Just click retry on the failed job in Cloud Build. This is fairly rare, as most failures involve a step failing after another has already succeeded.
 
 ### Divergence by exactly one commit.
 
-Find which commit caused the error.  This will usually be easy - cloud build lists the commit which triggered a build, so you can probably just use that one.  You need to set all the sync tags to the parent of that commit.  Say the commit which caused the error is `12345abc`.  You can find the parent of that commit with `git rev-parse 12345abc~` (note the `~` suffix).  Some of the sync tags are likely set to this value already.  For the remainder, simply perform a git push.  Assuming that the parent commit is `98765fed`, that would be, e.g. `git push origin 98765fed:tf-validator-sync`.
+Find which commit caused the error.  This will usually be easy - cloud build lists the commit which triggered a build, so you can probably just use that one.  You need to set all the sync tags to the parent of that commit.  Say the commit which caused the error is `12345abc`.  You can find the parent of that commit with `git rev-parse 12345abc~` (note the `~` suffix).  Some of the sync tags are likely set to this value already.  For the remainder, simply perform a git push.  Assuming that the parent commit is `98765fed`, that would be, e.g. `git push -f origin 98765fed:tf-validator-sync`.
 
 If you are unlucky, there may be open PRs - this only happens if the failure occurred during the ~5 second period surrounding the merging of one of the downstreams.  Close those PRs before proceeding to the final step.
 
@@ -103,6 +102,9 @@ The best approach is
 * Build the `downstream-generator` container locally, with the new Gemfile and Gemfile.lock.  This will involve hand-modifying the Dockerfile to use the local Gemfile/Gemfile.lock instead of wget from this repo's `main` branch.  You don't need to check in those changes.
 * When that container is built, and while nothing else is running in GCB (wait, if you need to), push the container to GCR, and as soon as possible afterwards, merge the dependency-changing PR.
 
+## Changes to cloud build yaml:
+If changes are made to `gcb-contributor-membership-checker.yml` or `gcb-community-checker.yml` they will not be reflected in presubmit runs for existing PRs without a rebase. This is because these build triggers are linked to pull request creation and not pushes to the PR branch. If changes are needed to these build files they will need to be made in a backwards-compatible manner. Note that changes to other files used by these triggers will be immediately reflected in all PRs, leading to a possible disconnect between the yaml files and the rest of the CI code.
+
 ## Historical Note: Design choices & tradeoffs
 * The downstream push doesn't wait for checks on its PRs against downstreams.  This may inconvenience some existing workflows which rely on the downstream PR checks.  This ensures that merge conflicts never come into play, since the downstreams never have dangling PRs, but it requires some up-front work to get those checks into the differ.  If a new check is introduced into the downstream Travis, we will need to introduce it into the terraform-tester container.
 * The downstream push is disconnected from the output of the differ (but runs the same code).  This means that the diff which is approved isn't guaranteed to be applied *exactly*, if for instance magic modules' behavior changes on main between diff generation and downstream push.  This is also intended to avoid merge conflicts by, effectively, rebasing each commit on top of main before final generation is done.

.ci/containers/README.md

@@ -3,17 +3,27 @@ The docker images located in this folder are used by multiple builds for magic m
 
 ## Naming Convention
 
-The images are named with the languages they contain and the images are versioned with tags that indicate the version of each language contained. eg: the image `go-ruby-python` with a tag of `1.11.5-2.6.0-2.7` indicates that the image has `go 1.11.5`, `ruby 2.6.0` and `python 2.7`.
+The images are named according to their use. We have a small number of images that get reused in multiple places, based around sets of requirements shared by different parts of the build pipeline. The images are:
 
-If there are multiple images with the same language version but different libraries (gems), a `v#` is appended to differentiate. eg: `1.11.5-2.6.0-2.7-v6`
+- `gcr.io/graphite-docker-images/bash-plus`
+- `gcr.io/graphite-docker-images/build-environment`
+- `gcr.io/graphite-docker-images/go-plus`
 
 ## Updating a docker image
-The Dockerfile should be updated, then the image rebuilt and pushed to the container registry stored at the `graphite-docker-images` GCP project. To update any of the images:
+
+Before you begin, set up Docker (including configuring it to [authenticate with gcloud](https://cloud.google.com/container-registry/docs/advanced-authentication#gcloud-helper)).
 
 1. Make changes to the Dockerfile
-2. Configure docker to use gcloud auth: `gcloud auth configure-docker`
-3. Build the image: `docker build . --tag gcr.io/graphite-docker-images/go-ruby-python`
-4. Find the new image's id: `docker images`
-5. Add the appropriate tag `docker tag ac37c0af8ce7 gcr.io/graphite-docker-images/go-ruby-python:1.11.5-2.6.0-2.7-v6`
-6. Push the image: `docker push gcr.io/graphite-docker-images/go-ruby-python:1.11.5-2.6.0-2.7-v6`
-7. Check the UI and ensure the new version is available and tagged at `latest`. It must be tagged `latest` for the Kokoro builds to get the correct version.
+2. Build the image with the `testing` tag:
+   ```bash
+   sudo docker build . --tag gcr.io/graphite-docker-images/bash-plus:testing
+   ```
+3. Push the image:
+   ```bash
+   sudo docker push gcr.io/graphite-docker-images/bash-plus:testing
+   ```
+4. Update cloudbuild yaml files to reference the image you just pushed by adding the `:testing` suffix
+5. Update files that will cause the cloudbuild yaml changes (and therefore your changes) to be exercised
+   - Tip: Modifying `mmv1/third_party/terraform/services/compute/metadata.go.erb` will trigger builds for TPG, TPGB, and TGC.
+6. Create a PR with these changes.
+7. Verify that the cloudbuild steps that should use your testing image _are_ using your testing image (in the Execution Details tab for the step.)

.ci/containers/build-environment/Dockerfile

@@ -1,5 +1,5 @@
 # Stage 1: Building Go dependencies
-FROM golang:1.20-bullseye AS builder
+FROM golang:1.21-bullseye AS builder
 
 # Set working directory
 WORKDIR /app
@@ -15,7 +15,7 @@ RUN go mod download
 FROM ruby:3.1-bullseye
 
 # golang
-COPY --from=golang:1.20-bullseye /usr/local/go /usr/local/go
+COPY --from=golang:1.21-bullseye /usr/local/go /usr/local/go
 ENV GOPATH /go
 ENV PATH /usr/local/go/bin:$PATH
 ENV PATH $GOPATH/bin:$PATH
@@ -36,10 +36,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends git openssh-cli
 RUN git config --global user.name "Modular Magician"
 RUN git config --global user.email "[email protected]"
 
-# Set up Github SSH cloning.
-RUN ssh-keyscan github.com >> /known_hosts
-RUN echo "UserKnownHostsFile /known_hosts" >> /etc/ssh/ssh_config
-
 RUN go install golang.org/x/tools/cmd/goimports@d088b475e3360caabc032aaee1dc66351d4e729a
 RUN go install github.com/github/[email protected]+incompatible
 

.ci/containers/go-plus/Dockerfile

@@ -1,5 +1,5 @@
 # Stage 1: Download go module cache for builds
-FROM golang:1.20-bullseye AS builder
+FROM golang:1.21-bullseye AS builder
 ENV GOCACHE=/go/cache
 
 RUN apt-get update && apt-get install -y unzip
@@ -12,18 +12,14 @@ WORKDIR /app1/magic-modules-main/.ci/magician
 RUN go build -o /dev/null .
 
 # Stage 2: Creating the final image
-FROM golang:1.20-bullseye
+FROM golang:1.21-bullseye
 SHELL ["/bin/bash", "-c"]
 ENV GOCACHE=/go/cache
 
 # Copy Go dependencies and Go build cache
 COPY --from=builder /go/pkg/mod /go/pkg/mod
 COPY --from=builder /go/cache /go/cache
 
-# Set up Github SSH cloning.
-RUN ssh-keyscan github.com >> /known_hosts
-RUN echo "UserKnownHostsFile /known_hosts" >> /etc/ssh/ssh_config
-
 RUN apt-get update && \
     apt-get install -y git jq unzip parallel curl && \
     echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \

.ci/gcb-community-checker.yml

@@ -61,7 +61,7 @@ steps:
   - name: 'gcr.io/graphite-docker-images/go-plus'
     entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
     id: community-checker
-    secretEnv: ["GITHUB_TOKEN", "GENERATE_DIFFS_TRIGGER"]
+    secretEnv: ["GITHUB_TOKEN_MAGIC_MODULES", "GENERATE_DIFFS_TRIGGER"]
     timeout: 8000s
     args:
         - "community-checker"
@@ -72,9 +72,10 @@ steps:
         - $_HEAD_BRANCH
         - $_BASE_BRANCH
 
+logsBucket: 'gs://cloudbuild-community-checker-logs'
 availableSecrets:
   secretManager:
-    - versionName: projects/673497134629/secrets/github-magician-token/versions/latest
-      env: GITHUB_TOKEN
+    - versionName: projects/673497134629/secrets/github-magician-token-generate-diffs-magic-modules/versions/latest
+      env: GITHUB_TOKEN_MAGIC_MODULES
     - versionName: projects/673497134629/secrets/ci-trigger-generate-diffs/versions/latest
       env: GENERATE_DIFFS_TRIGGER

.ci/gcb-contributor-membership-checker.yml

@@ -62,21 +62,18 @@ steps:
     entrypoint: "/workspace/.ci/scripts/go-plus/magician/exec.sh"
     id: contributor-membership-checker
     secretEnv:
-      ["GITHUB_TOKEN", "GENERATE_DIFFS_TRIGGER", "COMMUNITY_CHECKER_TRIGGER"]
+      ["GITHUB_TOKEN_MAGIC_MODULES", "GENERATE_DIFFS_TRIGGER", "COMMUNITY_CHECKER_TRIGGER"]
     timeout: 8000s
     args:
       - "membership-checker"
       - $_PR_NUMBER
       - $COMMIT_SHA
-      - $BRANCH_NAME
-      - $_HEAD_REPO_URL
-      - $_HEAD_BRANCH
-      - $_BASE_BRANCH
 
+logsBucket: 'gs://cloudbuild-membership-checker-logs'
 availableSecrets:
   secretManager:
-    - versionName: projects/673497134629/secrets/github-magician-token/versions/latest
-      env: GITHUB_TOKEN
+    - versionName: projects/673497134629/secrets/github-magician-token-generate-diffs-magic-modules/versions/latest
+      env: GITHUB_TOKEN_MAGIC_MODULES
     - versionName: projects/673497134629/secrets/ci-trigger-generate-diffs/versions/latest
       env: GENERATE_DIFFS_TRIGGER
     - versionName: projects/673497134629/secrets/ci-trigger-community-checker/versions/latest

.ci/gcb-generate-diffs-new.yml

@@ -72,7 +72,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/build-environment'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       id: tpg-head
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS"]
       waitFor: ["build-magician-binary"]
       env:
         - BASE_BRANCH=$_BASE_BRANCH
@@ -86,7 +86,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/build-environment'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       id: tpg-base
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS"]
       waitFor: ["build-magician-binary"]
       env:
         - BASE_BRANCH=$_BASE_BRANCH
@@ -99,7 +99,7 @@ steps:
 
     - name: 'gcr.io/graphite-docker-images/build-environment'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS"]
       id: tpgb-head
       waitFor: ["build-magician-binary"]
       env:
@@ -114,7 +114,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/build-environment'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       id: tpgb-base
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS"]
       waitFor: ["build-magician-binary"]
       env:
         - BASE_BRANCH=$_BASE_BRANCH
@@ -128,7 +128,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/build-environment'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       id: tgc-head
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS"]
       waitFor: ["build-magician-binary"]
       env:
         - BASE_BRANCH=$_BASE_BRANCH
@@ -142,7 +142,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/build-environment'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       id: tgc-base
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS"]
       waitFor: ["build-magician-binary"]
       env:
         - BASE_BRANCH=$_BASE_BRANCH
@@ -156,7 +156,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/build-environment'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       id: tf-oics-head
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS"]
       waitFor: ["build-magician-binary"]
       env:
         - BASE_BRANCH=$_BASE_BRANCH
@@ -170,7 +170,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/build-environment'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       id: tf-oics-base
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS"]
       waitFor: ["build-magician-binary"]
       env:
         - BASE_BRANCH=$_BASE_BRANCH
@@ -184,7 +184,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/go-plus'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       id: diff
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS", "GITHUB_TOKEN_MAGIC_MODULES"]
       args:
           - 'generate-comment'
       env:
@@ -198,7 +198,7 @@ steps:
       id: tgc-test
       allowFailure: true
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_MAGIC_MODULES"]
       waitFor: ["tpgb-head", "tpgb-base", "tgc-head", "tgc-base"]
       args:
         - 'test-tgc'
@@ -208,28 +208,29 @@ steps:
 
     - name: 'gcr.io/graphite-docker-images/go-plus'
       id: tgc-test-integration
-      entrypoint: '/workspace/.ci/scripts/go-plus/tgc-tester-integration/test_tgc_integration.sh'
+      entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
       allowFailure: true
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_MAGIC_MODULES"]
       waitFor: ["tpgb-head", "tpgb-base", "tgc-head", "tgc-base"]
       env:
         - TEST_PROJECT=$_VALIDATOR_TEST_PROJECT
         - TEST_FOLDER_ID=$_VALIDATOR_TEST_FOLDER
         - TEST_ANCESTRY=$_VALIDATOR_TEST_ANCESTRY
         - TEST_ORG_ID=$_VALIDATOR_TEST_ORG
       args:
-          - $_PR_NUMBER
-          - $COMMIT_SHA
-          - $BUILD_ID
-          - $PROJECT_ID
-          - "18"  # Build step
-          - terraform-google-conversion
+        - 'test-tgc-integration'
+        - $_PR_NUMBER
+        - $COMMIT_SHA
+        - $BUILD_ID
+        - $PROJECT_ID
+        - "18"  # Build step
+        - terraform-google-conversion
 
     - name: 'gcr.io/graphite-docker-images/go-plus'
       id: tpgb-test
       allowFailure: true
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_MAGIC_MODULES"]
       waitFor: ["tpgb-head", "tpgb-base"]
       args:
         - 'test-tpg'
@@ -242,7 +243,7 @@ steps:
       id: tpg-test
       allowFailure: true
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_MAGIC_MODULES"]
       waitFor: ["tpg-head", "tpg-base"]
       args:
         - 'test-tpg'
@@ -254,7 +255,7 @@ steps:
     - name: 'gcr.io/graphite-docker-images/go-plus'
       id: gcb-tpg-vcr-test
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
-      secretEnv: ["GITHUB_TOKEN", "GOOGLE_BILLING_ACCOUNT", "GOOGLE_CUST_ID", "GOOGLE_FIRESTORE_PROJECT", "GOOGLE_IDENTITY_USER", "GOOGLE_MASTER_BILLING_ACCOUNT", "GOOGLE_ORG", "GOOGLE_ORG_2", "GOOGLE_ORG_DOMAIN", "GOOGLE_PROJECT", "GOOGLE_PROJECT_NUMBER", "GOOGLE_SERVICE_ACCOUNT", "SA_KEY", "GOOGLE_PUBLIC_AVERTISED_PREFIX_DESCRIPTION", "GOOGLE_TPU_V2_VM_RUNTIME_VERSION"]
+      secretEnv: ["GITHUB_TOKEN_DOWNSTREAMS", "GITHUB_TOKEN_MAGIC_MODULES", "GOOGLE_BILLING_ACCOUNT", "GOOGLE_CUST_ID", "GOOGLE_FIRESTORE_PROJECT", "GOOGLE_IDENTITY_USER", "GOOGLE_MASTER_BILLING_ACCOUNT", "GOOGLE_ORG", "GOOGLE_ORG_2", "GOOGLE_ORG_DOMAIN", "GOOGLE_PROJECT", "GOOGLE_PROJECT_NUMBER", "GOOGLE_SERVICE_ACCOUNT", "SA_KEY", "GOOGLE_PUBLIC_AVERTISED_PREFIX_DESCRIPTION", "GOOGLE_TPU_V2_VM_RUNTIME_VERSION"]
       waitFor: ["diff"]
       env:
         - BASE_BRANCH=$_BASE_BRANCH
@@ -271,21 +272,26 @@ steps:
 
     - name: 'gcr.io/graphite-docker-images/go-plus'
       entrypoint: '/workspace/.ci/scripts/go-plus/magician/exec.sh'
-      secretEnv: ["GITHUB_TOKEN"]
+      secretEnv: ["GITHUB_TOKEN_MAGIC_MODULES"]
       waitFor: ["diff"]
       args:
           - 'request-service-reviewers'
           - $_PR_NUMBER
+      env:
+        - COMMIT_SHA=$COMMIT_SHA
 
 # Long timeout to enable waiting on VCR test
 timeout: 20000s
 options:
     machineType: 'N1_HIGHCPU_32'
 
+logsBucket: 'gs://cloudbuild-generate-diffs-logs'
 availableSecrets:
   secretManager:
-    - versionName: projects/673497134629/secrets/github-magician-token/versions/latest
-      env: GITHUB_TOKEN
+    - versionName: projects/673497134629/secrets/github-magician-token-generate-diffs-downstreams/versions/latest
+      env: GITHUB_TOKEN_DOWNSTREAMS
+    - versionName: projects/673497134629/secrets/github-magician-token-generate-diffs-magic-modules/versions/latest
+      env: GITHUB_TOKEN_MAGIC_MODULES
     - versionName: projects/673497134629/secrets/ci-test-billing-account/versions/latest
       env: GOOGLE_BILLING_ACCOUNT
     - versionName: projects/673497134629/secrets/ci-test-cust-id/versions/latest