Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherrypick AP cloud build stockout mitigation onto release-1.1 #580

Merged

Remove timeout from RayCluster CR apply; bump CB timeout to mitigate …

ce9bbc3
Select commit
Loading
Failed to load commit list.
Merged

Cherrypick AP cloud build stockout mitigation onto release-1.1 #580

Remove timeout from RayCluster CR apply; bump CB timeout to mitigate …
ce9bbc3
Select commit
Loading
Failed to load commit list.
Google Cloud Build / pr-review-trigger-ap-clusters (gke-ai-eco-dev) succeeded Apr 10, 2024 in 58m 32s

Summary

Build Information

Trigger pr-review-trigger-ap-clusters
Build dda42f08-e3a9-49bd-87f0-afb537a2f930
Start 2024-04-10T14:51:13-07:00
Duration 57m45.618s
Status SUCCESS

Steps

Step Status Duration
validate platform SUCCESS 1m1.142s
validate ray SUCCESS 19.008s
validate jupyterhub SUCCESS 18.857s
validate rag SUCCESS 20.74s
create gke cluster SUCCESS 10m59.302s
test ray cluster SUCCESS 8m1.818s
cleanup ray cluster SUCCESS 36.986s
test jupyterhub SUCCESS 2m7.482s
cleanup jupyterhub SUCCESS 40.278s
test rag SUCCESS 25m4.65s
cleanup rag SUCCESS 2m18.41s
cleanup gke cluster FAILURE 6m25.393s
check result SUCCESS 637ms

Details

starting build "dda42f08-e3a9-49bd-87f0-afb537a2f930"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/GoogleCloudPlatform/ai-on-gke
 * branch            ce9bbc3e2549ed5080790599a89b62e3fd8e405b -> FETCH_HEAD
HEAD is now at ce9bbc3 Remove timeout from RayCluster CR apply; bump CB timeout to mitigate stockouts (#576)
SETUPBUILD
BUILD
Starting Step #0 - "validate platform"
Step #0 - "validate platform": Pulling image: gcr.io/gke-ai-eco-dev/terraform
Step #0 - "validate platform": Using default tag: latest
Step #0 - "validate platform": latest: Pulling from gke-ai-eco-dev/terraform
Step #0 - "validate platform": ddf874abf16c: Pulling fs layer
Step #0 - "validate platform": ecfb911cf53f: Pulling fs layer
Step #0 - "validate platform": ce689bb70bd5: Pulling fs layer
Step #0 - "validate platform": 7d2b54998739: Pulling fs layer
Step #0 - "validate platform": b1c39b09da0f: Pulling fs layer
Step #0 - "validate platform": cc032834b94b: Pulling fs layer
Step #0 - "validate platform": 0279610fca41: Pulling fs layer
Step #0 - "validate platform": 6f125b563dda: Pulling fs layer
Step #0 - "validate platform": 7c8b6e22fcff: Pulling fs layer
Step #0 - "validate platform": faaaa1840355: Pulling fs layer
Step #0 - "validate platform": 60a28c2199b1: Pulling fs layer
Step #0 - "validate platform": 4c7171ac56db: Pulling fs layer
Step #0 - "validate platform": 372d85ec2f1f: Pulling fs layer
Step #0 - "validate platform": 633b411e5bd5: Pulling fs layer
Step #0 - "validate platform": b1c39b09da0f: Waiting
Step #0 - "validate platform": cc032834b94b: Waiting
Step #0 - "validate platform": 0279610fca41: Waiting
Step #0 - "validate platform": 4c7171ac56db: Waiting
Step #0 - "validate platform": 6f125b563dda: Waiting
Step #0 - "validate platform": 60a28c2199b1: Waiting
Step #0 - "validate platform": 372d85ec2f1f: Waiting
Step #0 - "validate platform": 7c8b6e22fcff: Waiting
Step #0 - "validate platform": 633b411e5bd5: Waiting
Step #0 - "validate platform": faaaa1840355: Waiting
Step #0 - "validate platform": ce689bb70bd5: Verifying Checksum
Step #0 - "validate platform": ce689bb70bd5: Download complete
Step #0 - "validate platform": 7d2b54998739: Download complete
Step #0 - "validate platform": ecfb911cf53f: Verifying Checksum
Step #0 - "validate platform": ecfb911cf53f: Download complete
Step #0 - "validate platform": ddf874abf16c: Verifying Checksum
Step #0 - "validate platform": ddf874abf16c: Download complete
Step #0 - "validate platform": 0279610fca41: Verifying Checksum
Step #0 - "validate platform": 0279610fca41: Download complete
Step #0 - "validate platform": 6f125b563dda: Verifying Checksum
Step #0 - "validate platform": 6f125b563dda: Download complete
Step #0 - "validate platform": 7c8b6e22fcff: Verifying Checksum
Step #0 - "validate platform": 7c8b6e22fcff: Download complete
Step #0 - "validate platform": faaaa1840355: Verifying Checksum
Step #0 - "validate platform": faaaa1840355: Download complete
Step #0 - "validate platform": cc032834b94b: Verifying Checksum
Step #0 - "validate platform": cc032834b94b: Download complete
Step #0 - "validate platform": 60a28c2199b1: Verifying Checksum
Step #0 - "validate platform": 60a28c2199b1: Download complete
Step #0 - "validate platform": 372d85ec2f1f: Download complete
Step #0 - "validate platform": 4c7171ac56db: Verifying Checksum
Step #0 - "validate platform": 4c7171ac56db: Download complete
Step #0 - "validate platform": ddf874abf16c: Pull complete
Step #0 - "validate platform": ecfb911cf53f: Pull complete
Step #0 - "validate platform": ce689bb70bd5: Pull complete
Step #0 - "validate platform": 7d2b54998739: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Verifying Checksum
Step #0 - "validate platform": 633b411e5bd5: Download complete
Step #0 - "validate platform": b1c39b09da0f: Verifying Checksum
Step #0 - "validate platform": b1c39b09da0f: Download complete
Step #0 - "validate platform": b1c39b09da0f: Pull complete
Step #0 - "validate platform": cc032834b94b: Pull complete
Step #0 - "validate platform": 0279610fca41: Pull complete
Step #0 - "validate platform": 6f125b563dda: Pull complete
Step #0 - "validate platform": 7c8b6e22fcff: Pull complete
Step #0 - "validate platform": faaaa1840355: Pull complete
Step #0 - "validate platform": 60a28c2199b1: Pull complete
Step #0 - "validate platform": 4c7171ac56db: Pull complete
Step #0 - "validate platform": 372d85ec2f1f: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Pull complete
Step #0 - "validate platform": Digest: sha256:202db292b551d530ff51210759dd25971cb64b5afe675a5616e0c244d48a452b
Step #0 - "validate platform": Status: Downloaded newer image for gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing the backend...
Step #0 - "validate platform": Initializing modules...
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for cloud-nat...
Step #0 - "validate platform": - cloud-nat in .terraform/modules/cloud-nat
Step #0 - "validate platform": - custom-network in ../modules/gcp-network
Step #0 - "validate platform": - private-gke-autopilot-cluster in ../modules/gke-autopilot-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for private-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - private-gke-autopilot-cluster.gke in .terraform/modules/private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #0 - "validate platform": - private-gke-standard-cluster in ../modules/gke-standard-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for private-gke-standard-cluster.gke...
Step #0 - "validate platform": - private-gke-standard-cluster.gke in .terraform/modules/private-gke-standard-cluster.gke/modules/private-cluster
Step #0 - "validate platform": - public-gke-autopilot-cluster in ../modules/gke-autopilot-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for public-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - public-gke-autopilot-cluster.gke in .terraform/modules/public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #0 - "validate platform": - public-gke-standard-cluster in ../modules/gke-standard-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for public-gke-standard-cluster.gke...
Step #0 - "validate platform": - public-gke-standard-cluster.gke in .terraform/modules/public-gke-standard-cluster.gke
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing provider plugins...
Step #0 - "validate platform": - Finding hashicorp/google versions matching ">= 4.51.0, >= 5.0.0, < 6.0.0"...
Step #0 - "validate platform": - Finding hashicorp/google-beta versions matching ">= 5.0.0, < 6.0.0"...
Step #0 - "validate platform": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #0 - "validate platform": - Finding hashicorp/kubernetes versions matching "~> 2.10, 2.18.1"...
Step #0 - "validate platform": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #0 - "validate platform": - Installing hashicorp/random v3.6.0...
Step #0 - "validate platform": - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/google v5.24.0...
Step #0 - "validate platform": - Installed hashicorp/google v5.24.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/google-beta v5.24.0...
Step #0 - "validate platform": - Installed hashicorp/google-beta v5.24.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/helm v2.8.0...
Step #0 - "validate platform": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/kubernetes v2.18.1...
Step #0 - "validate platform": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #0 - "validate platform": selections it made above. Include this file in your version control repository
Step #0 - "validate platform": so that Terraform can guarantee to make the same selections by default when
Step #0 - "validate platform": you run "terraform init" in the future.
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has been successfully initialized!
Step #0 - "validate platform": 
Step #0 - "validate platform": You may now begin working with Terraform. Try running "terraform plan" to see
Step #0 - "validate platform": any changes that are required for your infrastructure. All Terraform commands
Step #0 - "validate platform": should now work.
Step #0 - "validate platform": 
Step #0 - "validate platform": If you ever set or change modules or backend configuration for Terraform,
Step #0 - "validate platform": rerun this command to reinitialize your working directory. If you forget, other
Step #0 - "validate platform": commands will detect it and remind you to do so if necessary.
Step #0 - "validate platform": Success! The configuration is valid.
Step #0 - "validate platform": 
Finished Step #0 - "validate platform"
Starting Step #1 - "validate ray"
Starting Step #3 - "validate rag"
Starting Step #2 - "validate jupyterhub"
Step #1 - "validate ray": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #3 - "validate rag": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #2 - "validate jupyterhub": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing the backend...
Step #1 - "validate ray": Initializing modules...
Step #1 - "validate ray": - gcs in ../../modules/gcs
Step #1 - "validate ray": - infra in ../../infrastructure
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing the backend...
Step #2 - "validate jupyterhub": Initializing modules...
Step #2 - "validate jupyterhub": - gcs in ../../modules/gcs
Step #2 - "validate jupyterhub": - infra in ../../infrastructure
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing the backend...
Step #3 - "validate rag": Initializing modules...
Step #3 - "validate rag": - cloudsql in ../../modules/cloudsql
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/sql-db/google 20.0.0 for cloudsql.cloudsql...
Step #1 - "validate ray": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #1 - "validate ray": - infra.custom-network in ../../modules/gcp-network
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #2 - "validate jupyterhub": - infra.custom-network in ../../modules/gcp-network
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - cloudsql.cloudsql in .terraform/modules/cloudsql.cloudsql/modules/postgresql
Step #3 - "validate rag": - frontend in frontend
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for frontend.frontend-workload-identity...
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #1 - "validate ray": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #1 - "validate ray": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #1 - "validate ray": - kuberay-cluster in ../../modules/kuberay-cluster
Step #1 - "validate ray": - kuberay-cluster.iap_auth in ../../modules/iap
Step #1 - "validate ray": - kuberay-logging in ../../modules/kuberay-logging
Step #1 - "validate ray": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #1 - "validate ray": - kuberay-operator in ../../modules/kuberay-operator
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-operator.kuberay-workload-identity...
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-standard-cluster.gke...
Step #3 - "validate rag": - frontend.frontend-workload-identity in .terraform/modules/frontend.frontend-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for frontend.frontend-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #2 - "validate jupyterhub": - jupyterhub in ../../modules/jupyter
Step #2 - "validate jupyterhub": - jupyterhub.iap_auth in ../../modules/iap
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa in .terraform/modules/frontend.frontend-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/frontend.frontend-workload-identity.annotate-sa
Step #3 - "validate rag": - frontend.iap_auth in ../../modules/iap
Step #3 - "validate rag": - gcs in ../../modules/gcs
Step #3 - "validate rag": - inference-server in ../../tutorials-and-examples/hf-tgi
Step #3 - "validate rag": - infra in ../../infrastructure
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #1 - "validate ray": - kuberay-operator.kuberay-workload-identity in .terraform/modules/kuberay-operator.kuberay-workload-identity/modules/workload-identity
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for kuberay-operator.kuberay-workload-identity.annotate-sa...
Step #3 - "validate rag": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #3 - "validate rag": - infra.custom-network in ../../modules/gcp-network
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #1 - "validate ray": - kuberay-operator.kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-operator.kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #1 - "validate ray": - kuberay-operator.kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-operator.kuberay-workload-identity.annotate-sa
Step #1 - "validate ray": - namespace in ../../modules/kubernetes-namespace
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #2 - "validate jupyterhub": - namespace in ../../modules/kubernetes-namespace
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #1 - "validate ray": - project-services in .terraform/modules/project-services/modules/project_services
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing provider plugins...
Step #1 - "validate ray": - terraform.io/builtin/terraform is built in to Terraform
Step #1 - "validate ray": - Finding hashicorp/time versions matching "0.11.1"...
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #3 - "validate rag": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-standard-cluster.gke...
Step #1 - "validate ray": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #3 - "validate rag": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #1 - "validate ray": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #3 - "validate rag": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.0.0, < 6.0.0"...
Step #3 - "validate rag": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #3 - "validate rag": - jupyterhub in ../../modules/jupyter
Step #3 - "validate rag": - jupyterhub.iap_auth in ../../modules/iap
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #1 - "validate ray": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.0.0, < 6.0.0"...
Step #1 - "validate ray": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #3 - "validate rag": - kuberay-cluster in ../../modules/kuberay-cluster
Step #3 - "validate rag": - kuberay-cluster.iap_auth in ../../modules/iap
Step #3 - "validate rag": - kuberay-logging in ../../modules/kuberay-logging
Step #3 - "validate rag": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #3 - "validate rag": - kuberay-operator in ../../modules/kuberay-operator
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-operator.kuberay-workload-identity...
Step #1 - "validate ray": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #2 - "validate jupyterhub": - project-services in .terraform/modules/project-services/modules/project_services
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing provider plugins...
Step #2 - "validate jupyterhub": - terraform.io/builtin/terraform is built in to Terraform
Step #2 - "validate jupyterhub": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #3 - "validate rag": - kuberay-operator.kuberay-workload-identity in .terraform/modules/kuberay-operator.kuberay-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for kuberay-operator.kuberay-workload-identity.annotate-sa...
Step #3 - "validate rag": - kuberay-operator.kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-operator.kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #2 - "validate jupyterhub": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #3 - "validate rag": - kuberay-operator.kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-operator.kuberay-workload-identity.annotate-sa
Step #3 - "validate rag": - namespace in ../../modules/kubernetes-namespace
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #2 - "validate jupyterhub": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #1 - "validate ray": - Installing hashicorp/external v2.3.3...
Step #2 - "validate jupyterhub": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #2 - "validate jupyterhub": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.0.0, < 6.0.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.0.0, < 6.0.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #2 - "validate jupyterhub": - Installing hashicorp/helm v2.8.0...
Step #1 - "validate ray": - Installed hashicorp/external v2.3.3 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/random v3.6.0...
Step #3 - "validate rag": - project-services in .terraform/modules/project-services/modules/project_services
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing provider plugins...
Step #3 - "validate rag": - terraform.io/builtin/terraform is built in to Terraform
Step #3 - "validate rag": - Finding hashicorp/null versions matching ">= 2.1.0, ~> 3.1"...
Step #3 - "validate rag": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #3 - "validate rag": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.0.0, >= 5.12.0, < 6.0.0"...
Step #3 - "validate rag": - Finding hashicorp/time versions matching "0.11.1"...
Step #1 - "validate ray": - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
Step #3 - "validate rag": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0, ~> 3.1"...
Step #3 - "validate rag": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #1 - "validate ray": - Installing hashicorp/google v5.24.0...
Step #3 - "validate rag": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.0.0, >= 5.12.0, < 6.0.0"...
Step #3 - "validate rag": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #3 - "validate rag": - Installing hashicorp/kubernetes v2.18.1...
Step #2 - "validate jupyterhub": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/random v3.6.0...
Step #2 - "validate jupyterhub": - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/null v3.2.2...
Step #3 - "validate rag": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/null v3.2.2...
Step #2 - "validate jupyterhub": - Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/external v2.3.3...
Step #3 - "validate rag": - Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/google v5.24.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/helm v2.8.0...
Step #2 - "validate jupyterhub": - Installed hashicorp/external v2.3.3 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/google-beta v5.24.0...
Step #2 - "validate jupyterhub": - Installing hashicorp/kubernetes v2.18.1...
Step #3 - "validate rag": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/google v5.24.0...
Step #2 - "validate jupyterhub": - Installing hashicorp/google v5.24.0...
Step #1 - "validate ray": - Installed hashicorp/google-beta v5.24.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/helm v2.8.0...
Step #3 - "validate rag": - Installed hashicorp/google v5.24.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/google v5.24.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/time v0.11.1...
Step #2 - "validate jupyterhub": - Installing hashicorp/google-beta v5.24.0...
Step #1 - "validate ray": - Installing hashicorp/kubernetes v2.18.1...
Step #3 - "validate rag": - Installed hashicorp/time v0.11.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/random v3.6.0...
Step #3 - "validate rag": - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/external v2.3.3...
Step #1 - "validate ray": - Installing hashicorp/time v0.11.1...
Step #3 - "validate rag": - Installed hashicorp/external v2.3.3 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/time v0.11.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/google-beta v5.24.0...
Step #1 - "validate ray": - Installing hashicorp/null v3.2.2...
Step #2 - "validate jupyterhub": - Installed hashicorp/google-beta v5.24.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #2 - "validate jupyterhub": selections it made above. Include this file in your version control repository
Step #2 - "validate jupyterhub": so that Terraform can guarantee to make the same selections by default when
Step #2 - "validate jupyterhub": you run "terraform init" in the future.
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Terraform has been successfully initialized!
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Y
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/dda42f08-e3a9-49bd-87f0-afb537a2f930?project=142966328212.]
...
nup gke cluster":                   - enable_integrity_monitoring = true -> null
Step #11 - "cleanup gke cluster":                   - enable_secure_boot          = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - workload_metadata_config {
Step #11 - "cleanup gke cluster":                   - mode = "GKE_METADATA" -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - placement_policy {}
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - upgrade_settings {
Step #11 - "cleanup gke cluster":               - max_surge       = 1 -> null
Step #11 - "cleanup gke cluster":               - max_unavailable = 0 -> null
Step #11 - "cleanup gke cluster":               - strategy        = "SURGE" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster":       - node_pool {
Step #11 - "cleanup gke cluster":           - initial_node_count          = 0 -> null
Step #11 - "cleanup gke cluster":           - instance_group_urls         = [] -> null
Step #11 - "cleanup gke cluster":           - managed_instance_group_urls = [] -> null
Step #11 - "cleanup gke cluster":           - max_pods_per_node           = 16 -> null
Step #11 - "cleanup gke cluster":           - name                        = "nap-s2myqnus" -> null
Step #11 - "cleanup gke cluster":           - node_count                  = 0 -> null
Step #11 - "cleanup gke cluster":           - node_locations              = [
Step #11 - "cleanup gke cluster":               - "us-east4-a",
Step #11 - "cleanup gke cluster":               - "us-east4-c",
Step #11 - "cleanup gke cluster":             ] -> null
Step #11 - "cleanup gke cluster":           - version                     = "1.28.7-gke.1026000" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - autoscaling {
Step #11 - "cleanup gke cluster":               - location_policy      = "BALANCED" -> null
Step #11 - "cleanup gke cluster":               - max_node_count       = 1000 -> null
Step #11 - "cleanup gke cluster":               - min_node_count       = 0 -> null
Step #11 - "cleanup gke cluster":               - total_max_node_count = 0 -> null
Step #11 - "cleanup gke cluster":               - total_min_node_count = 0 -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - management {
Step #11 - "cleanup gke cluster":               - auto_repair  = true -> null
Step #11 - "cleanup gke cluster":               - auto_upgrade = true -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - network_config {
Step #11 - "cleanup gke cluster":               - create_pod_range     = false -> null
Step #11 - "cleanup gke cluster":               - enable_private_nodes = false -> null
Step #11 - "cleanup gke cluster":               - pod_ipv4_cidr_block  = "10.103.0.0/17" -> null
Step #11 - "cleanup gke cluster":               - pod_range            = "gke-ml-ce9bbc3-580-dda42f08-cluster-pods-f6e26001" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - node_config {
Step #11 - "cleanup gke cluster":               - disk_size_gb                = 250 -> null
Step #11 - "cleanup gke cluster":               - disk_type                   = "pd-balanced" -> null
Step #11 - "cleanup gke cluster":               - effective_taints            = [
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - effect = "NO_SCHEDULE"
Step #11 - "cleanup gke cluster":                       - key    = "cloud.google.com/compute-class"
Step #11 - "cleanup gke cluster":                       - value  = "Accelerator"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - effect = "NO_SCHEDULE"
Step #11 - "cleanup gke cluster":                       - key    = "cloud.google.com/gke-ephemeral-storage-local-ssd"
Step #11 - "cleanup gke cluster":                       - value  = "true"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - effect = "NO_SCHEDULE"
Step #11 - "cleanup gke cluster":                       - key    = "nvidia.com/gpu"
Step #11 - "cleanup gke cluster":                       - value  = "present"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                 ] -> null
Step #11 - "cleanup gke cluster":               - enable_confidential_storage = false -> null
Step #11 - "cleanup gke cluster":               - guest_accelerator           = [
Step #11 - "cleanup gke cluster":                   - {
Step #11 - "cleanup gke cluster":                       - count                          = 1
Step #11 - "cleanup gke cluster":                       - gpu_driver_installation_config = [
Step #11 - "cleanup gke cluster":                           - {
Step #11 - "cleanup gke cluster":                               - gpu_driver_version = "INSTALLATION_DISABLED"
Step #11 - "cleanup gke cluster":                             },
Step #11 - "cleanup gke cluster":                         ]
Step #11 - "cleanup gke cluster":                       - gpu_partition_size             = ""
Step #11 - "cleanup gke cluster":                       - gpu_sharing_config             = []
Step #11 - "cleanup gke cluster":                       - type                           = "nvidia-l4"
Step #11 - "cleanup gke cluster":                     },
Step #11 - "cleanup gke cluster":                 ] -> null
Step #11 - "cleanup gke cluster":               - image_type                  = "COS_CONTAINERD" -> null
Step #11 - "cleanup gke cluster":               - labels                      = {
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/compute-class"         = "Accelerator"
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/gke-accelerator-count" = "1"
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/pod-isolation"         = "10100m"
Step #11 - "cleanup gke cluster":                   - "cloud.google.com/pod-slots"             = "1"
Step #11 - "cleanup gke cluster":                 } -> null
Step #11 - "cleanup gke cluster":               - local_ssd_count             = 0 -> null
Step #11 - "cleanup gke cluster":               - logging_variant             = "DEFAULT" -> null
Step #11 - "cleanup gke cluster":               - machine_type                = "g2-standard-12" -> null
Step #11 - "cleanup gke cluster":               - metadata                    = {
Step #11 - "cleanup gke cluster":                   - "disable-legacy-endpoints" = "true"
Step #11 - "cleanup gke cluster":                 } -> null
Step #11 - "cleanup gke cluster":               - oauth_scopes                = [
Step #11 - "cleanup gke cluster":                   - "https://www.googleapis.com/auth/cloud-platform",
Step #11 - "cleanup gke cluster":                   - "https://www.googleapis.com/auth/userinfo.email",
Step #11 - "cleanup gke cluster":                 ] -> null
Step #11 - "cleanup gke cluster":               - preemptible                 = false -> null
Step #11 - "cleanup gke cluster":               - resource_labels             = {} -> null
Step #11 - "cleanup gke cluster":               - resource_manager_tags       = {} -> null
Step #11 - "cleanup gke cluster":               - service_account             = "tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":               - spot                        = false -> null
Step #11 - "cleanup gke cluster":               - tags                        = [] -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - ephemeral_storage_config {
Step #11 - "cleanup gke cluster":                   - local_ssd_count = 1 -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - reservation_affinity {
Step #11 - "cleanup gke cluster":                   - consume_reservation_type = "NO_RESERVATION" -> null
Step #11 - "cleanup gke cluster":                   - values                   = [] -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - shielded_instance_config {
Step #11 - "cleanup gke cluster":                   - enable_integrity_monitoring = true -> null
Step #11 - "cleanup gke cluster":                   - enable_secure_boot          = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - workload_metadata_config {
Step #11 - "cleanup gke cluster":                   - mode = "GKE_METADATA" -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - placement_policy {}
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - upgrade_settings {
Step #11 - "cleanup gke cluster":               - max_surge       = 1 -> null
Step #11 - "cleanup gke cluster":               - max_unavailable = 0 -> null
Step #11 - "cleanup gke cluster":               - strategy        = "SURGE" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - node_pool_defaults {
Step #11 - "cleanup gke cluster":           - node_config_defaults {
Step #11 - "cleanup gke cluster":               - logging_variant = "DEFAULT" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":               - gcfs_config {
Step #11 - "cleanup gke cluster":                   - enabled = true -> null
Step #11 - "cleanup gke cluster":                 }
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - notification_config {
Step #11 - "cleanup gke cluster":           - pubsub {
Step #11 - "cleanup gke cluster":               - enabled = false -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - pod_security_policy_config {
Step #11 - "cleanup gke cluster":           - enabled = false -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - private_cluster_config {
Step #11 - "cleanup gke cluster":           - enable_private_endpoint = false -> null
Step #11 - "cleanup gke cluster":           - enable_private_nodes    = false -> null
Step #11 - "cleanup gke cluster":           - private_endpoint        = "10.128.0.2" -> null
Step #11 - "cleanup gke cluster":           - public_endpoint         = "34.86.116.9" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - master_global_access_config {
Step #11 - "cleanup gke cluster":               - enabled = false -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - protect_config {
Step #11 - "cleanup gke cluster":           - workload_vulnerability_mode = "DISABLED" -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - workload_config {
Step #11 - "cleanup gke cluster":               - audit_mode = "DISABLED" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - release_channel {
Step #11 - "cleanup gke cluster":           - channel = "REGULAR" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - security_posture_config {
Step #11 - "cleanup gke cluster":           - mode               = "DISABLED" -> null
Step #11 - "cleanup gke cluster":           - vulnerability_mode = "VULNERABILITY_DISABLED" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - service_external_ips_config {
Step #11 - "cleanup gke cluster":           - enabled = false -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - timeouts {
Step #11 - "cleanup gke cluster":           - create = "45m" -> null
Step #11 - "cleanup gke cluster":           - delete = "45m" -> null
Step #11 - "cleanup gke cluster":           - update = "45m" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - vertical_pod_autoscaling {
Step #11 - "cleanup gke cluster":           - enabled = true -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - workload_identity_config {
Step #11 - "cleanup gke cluster":           - workload_pool = "gke-ai-eco-dev.svc.id.goog" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-nodeService_account" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYVxbxlUIc=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/container.nodeServiceAccount/serviceAccount:tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/container.nodeServiceAccount" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_service_account" "cluster_service_account" {
Step #11 - "cleanup gke cluster":       - account_id   = "tf-gke-ml-ce9bbc3-580--zmiv" -> null
Step #11 - "cleanup gke cluster":       - disabled     = false -> null
Step #11 - "cleanup gke cluster":       - display_name = "Terraform-managed service account for cluster ml-ce9bbc3-580-dda42f08-cluster" -> null
Step #11 - "cleanup gke cluster":       - email        = "tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - id           = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member       = "serviceAccount:tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - name         = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project      = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - unique_id    = "107611923864321847169" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix will be destroyed
Step #11 - "cleanup gke cluster":   - resource "random_string" "cluster_service_account_suffix" {
Step #11 - "cleanup gke cluster":       - id          = "zmiv" -> null
Step #11 - "cleanup gke cluster":       - length      = 4 -> null
Step #11 - "cleanup gke cluster":       - lower       = true -> null
Step #11 - "cleanup gke cluster":       - min_lower   = 0 -> null
Step #11 - "cleanup gke cluster":       - min_numeric = 0 -> null
Step #11 - "cleanup gke cluster":       - min_special = 0 -> null
Step #11 - "cleanup gke cluster":       - min_upper   = 0 -> null
Step #11 - "cleanup gke cluster":       - number      = true -> null
Step #11 - "cleanup gke cluster":       - numeric     = true -> null
Step #11 - "cleanup gke cluster":       - result      = "zmiv" -> null
Step #11 - "cleanup gke cluster":       - special     = false -> null
Step #11 - "cleanup gke cluster":       - upper       = false -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Plan: 0 to add, 0 to change, 8 to destroy.
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Changes to Outputs:
Step #11 - "cleanup gke cluster":   - ca_certificate   = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - cluster_location = "us-east4" -> null
Step #11 - "cleanup gke cluster":   - cluster_name     = "ml-ce9bbc3-580-dda42f08-cluster" -> null
Step #11 - "cleanup gke cluster":   - endpoint         = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - private_cluster  = false -> null
Step #11 - "cleanup gke cluster":   - project_id       = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Destroying... [id=projects/gke-ai-eco-dev/locations/us-east4/clusters/ml-ce9bbc3-580-dda42f08-cluster]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destroying... [id=gke-ai-eco-dev/roles/container.nodeServiceAccount/serviceAccount:tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 1m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 2m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 2m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 2m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 2m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 2m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 2m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 3m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 3m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 3m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 3m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 3m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 3m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 4m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 4m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 4m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 4m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 4m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 4m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 5m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 5m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 5m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-ce9bbc3-580-dda42f08-cluster, 5m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_container_cluster.primary: Destruction complete after 5m33s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destroying... [id=projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-ce9bbc3-580--zmiv@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix: Destroying... [id=zmiv]
Step #11 - "cleanup gke cluster": module.public-gke-autopilot-cluster[0].module.gke.random_string.cluster_service_account_suffix: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_service_networking_connection.private_service_access[0]: Destroying... [id=https%3A%2F%2Fwww.googleapis.com%2Fcompute%2Fv1%2Fprojects%2Fgke-ai-eco-dev%2Fglobal%2Fnetworks%2Fml-ce9bbc3-580-dda42f08-true:servicenetworking.googleapis.com]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_service_networking_connection.private_service_access[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-ce9bbc3-580-dda42f08-true"]: Destroying... [id=projects/gke-ai-eco-dev/regions/us-east4/subnetworks/ml-ce9bbc3-580-dda42f08-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Destroying... [id=projects/gke-ai-eco-dev/global/addresses/google-managed-services-ml-ce9bbc3-580-dda42f08-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-ce9bbc3-580-dda42f08-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-ce9bbc3-580-dda42f08-true, 10s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Still destroying... [id=projects/gke-ai-eco-dev/global/addresse...-services-ml-ce9bbc3-580-dda42f08-true, 10s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Destruction complete after 11s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-ce9bbc3-580-dda42f08-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-ce9bbc3-580-dda42f08-true, 20s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-ce9bbc3-580-dda42f08-true"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...bnetworks/ml-ce9bbc3-580-dda42f08-true, 30s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-ce9bbc3-580-dda42f08-true"]: Destruction complete after 31s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_network.network: Destroying... [id=projects/gke-ai-eco-dev/global/networks/ml-ce9bbc3-580-dda42f08-true]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_network.network: Still destroying... [id=projects/gke-ai-eco-dev/global/networks/ml-ce9bbc3-580-dda42f08-true, 10s elapsed]
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Error: Error waiting for Deleting Network: The network resource 'projects/gke-ai-eco-dev/global/networks/ml-ce9bbc3-580-dda42f08-true' is already being used by 'projects/gke-ai-eco-dev/global/firewalls/ml-ce9bbc3-580-dda42f08-true-ygmk6bsxwqwslgva56uurekb'
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": 
Finished Step #11 - "cleanup gke cluster"
ERROR: step exited with non-zero status: 1
Starting Step #12 - "check result"
Step #12 - "check result": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Finished Step #12 - "check result"
PUSH
DONE

Build Log: https://console.cloud.google.com/cloud-build/builds;region=us-central1/dda42f08-e3a9-49bd-87f0-afb537a2f930?project=142966328212