Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add security context to Ray clusters #281

Merged
merged 9 commits into from
Mar 8, 2024

Merge branch 'main' into ray_security_context

f412967
Select commit
Loading
Failed to load commit list.
Merged

Add security context to Ray clusters #281

Merge branch 'main' into ray_security_context
f412967
Select commit
Loading
Failed to load commit list.
Google Cloud Build / pr-review-trigger (gke-ai-eco-dev) succeeded Mar 8, 2024 in 45m 6s

Summary

Build Information

Trigger pr-review-trigger
Build 1a17ac6c-140e-4f89-b838-453b2f3e1349
Start 2024-03-08T10:37:18-08:00
Duration 44m31.464s
Status SUCCESS

Steps

Step Status Duration
validate platform SUCCESS 48.146s
validate ray SUCCESS 15.507s
validate jupyterhub SUCCESS 15.51s
validate rag SUCCESS 15.513s
create gke cluster SUCCESS 9m32.843s
test ray cluster SUCCESS 2m52.509s
cleanup ray cluster SUCCESS 23.23s
test jupyterhub SUCCESS 2m4.88s
cleanup jupyterhub SUCCESS 49.165s
test rag SUCCESS 14m18.856s
cleanup rag SUCCESS 1m54.392s
cleanup gke cluster SUCCESS 11m26.348s
check result SUCCESS 670ms

Details

starting build "1a17ac6c-140e-4f89-b838-453b2f3e1349"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint: 	git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/GoogleCloudPlatform/ai-on-gke
 * branch            f412967be7a4d887d46c1c554d1c4d46f6514016 -> FETCH_HEAD
HEAD is now at f412967 Merge branch 'main' into ray_security_context
SETUPBUILD
BUILD
Starting Step #0 - "validate platform"
Step #0 - "validate platform": Pulling image: gcr.io/gke-ai-eco-dev/terraform
Step #0 - "validate platform": Using default tag: latest
Step #0 - "validate platform": latest: Pulling from gke-ai-eco-dev/terraform
Step #0 - "validate platform": ddf874abf16c: Pulling fs layer
Step #0 - "validate platform": ecfb911cf53f: Pulling fs layer
Step #0 - "validate platform": ce689bb70bd5: Pulling fs layer
Step #0 - "validate platform": 7d2b54998739: Pulling fs layer
Step #0 - "validate platform": b1c39b09da0f: Pulling fs layer
Step #0 - "validate platform": cc032834b94b: Pulling fs layer
Step #0 - "validate platform": 0279610fca41: Pulling fs layer
Step #0 - "validate platform": 6f125b563dda: Pulling fs layer
Step #0 - "validate platform": 7c8b6e22fcff: Pulling fs layer
Step #0 - "validate platform": faaaa1840355: Pulling fs layer
Step #0 - "validate platform": 60a28c2199b1: Pulling fs layer
Step #0 - "validate platform": 4c7171ac56db: Pulling fs layer
Step #0 - "validate platform": 372d85ec2f1f: Pulling fs layer
Step #0 - "validate platform": 633b411e5bd5: Pulling fs layer
Step #0 - "validate platform": b1c39b09da0f: Waiting
Step #0 - "validate platform": cc032834b94b: Waiting
Step #0 - "validate platform": 0279610fca41: Waiting
Step #0 - "validate platform": 6f125b563dda: Waiting
Step #0 - "validate platform": 7c8b6e22fcff: Waiting
Step #0 - "validate platform": faaaa1840355: Waiting
Step #0 - "validate platform": 60a28c2199b1: Waiting
Step #0 - "validate platform": 4c7171ac56db: Waiting
Step #0 - "validate platform": 372d85ec2f1f: Waiting
Step #0 - "validate platform": 7d2b54998739: Waiting
Step #0 - "validate platform": 633b411e5bd5: Waiting
Step #0 - "validate platform": ce689bb70bd5: Verifying Checksum
Step #0 - "validate platform": ce689bb70bd5: Download complete
Step #0 - "validate platform": 7d2b54998739: Verifying Checksum
Step #0 - "validate platform": 7d2b54998739: Download complete
Step #0 - "validate platform": ecfb911cf53f: Verifying Checksum
Step #0 - "validate platform": ecfb911cf53f: Download complete
Step #0 - "validate platform": ddf874abf16c: Verifying Checksum
Step #0 - "validate platform": ddf874abf16c: Download complete
Step #0 - "validate platform": 0279610fca41: Verifying Checksum
Step #0 - "validate platform": 0279610fca41: Download complete
Step #0 - "validate platform": 6f125b563dda: Verifying Checksum
Step #0 - "validate platform": 6f125b563dda: Download complete
Step #0 - "validate platform": 7c8b6e22fcff: Download complete
Step #0 - "validate platform": cc032834b94b: Verifying Checksum
Step #0 - "validate platform": cc032834b94b: Download complete
Step #0 - "validate platform": faaaa1840355: Verifying Checksum
Step #0 - "validate platform": faaaa1840355: Download complete
Step #0 - "validate platform": 60a28c2199b1: Verifying Checksum
Step #0 - "validate platform": 60a28c2199b1: Download complete
Step #0 - "validate platform": 372d85ec2f1f: Verifying Checksum
Step #0 - "validate platform": 372d85ec2f1f: Download complete
Step #0 - "validate platform": ddf874abf16c: Pull complete
Step #0 - "validate platform": 4c7171ac56db: Verifying Checksum
Step #0 - "validate platform": 4c7171ac56db: Download complete
Step #0 - "validate platform": ecfb911cf53f: Pull complete
Step #0 - "validate platform": ce689bb70bd5: Pull complete
Step #0 - "validate platform": 7d2b54998739: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Download complete
Step #0 - "validate platform": b1c39b09da0f: Verifying Checksum
Step #0 - "validate platform": b1c39b09da0f: Download complete
Step #0 - "validate platform": b1c39b09da0f: Pull complete
Step #0 - "validate platform": cc032834b94b: Pull complete
Step #0 - "validate platform": 0279610fca41: Pull complete
Step #0 - "validate platform": 6f125b563dda: Pull complete
Step #0 - "validate platform": 7c8b6e22fcff: Pull complete
Step #0 - "validate platform": faaaa1840355: Pull complete
Step #0 - "validate platform": 60a28c2199b1: Pull complete
Step #0 - "validate platform": 4c7171ac56db: Pull complete
Step #0 - "validate platform": 372d85ec2f1f: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Pull complete
Step #0 - "validate platform": Digest: sha256:202db292b551d530ff51210759dd25971cb64b5afe675a5616e0c244d48a452b
Step #0 - "validate platform": Status: Downloaded newer image for gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing the backend...
Step #0 - "validate platform": Initializing modules...
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for cloud-nat...
Step #0 - "validate platform": - cloud-nat in .terraform/modules/cloud-nat
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/network/google 8.0.0 for custom-network...
Step #0 - "validate platform": - custom-network in .terraform/modules/custom-network
Step #0 - "validate platform": - custom-network.firewall_rules in .terraform/modules/custom-network/modules/firewall-rules
Step #0 - "validate platform": - custom-network.routes in .terraform/modules/custom-network/modules/routes
Step #0 - "validate platform": - custom-network.subnets in .terraform/modules/custom-network/modules/subnets
Step #0 - "validate platform": - custom-network.vpc in .terraform/modules/custom-network/modules/vpc
Step #0 - "validate platform": - private-gke-autopilot-cluster in ../modules/gke-autopilot-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for private-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - private-gke-autopilot-cluster.gke in .terraform/modules/private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #0 - "validate platform": - private-gke-standard-cluster in ../modules/gke-standard-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for private-gke-standard-cluster.gke...
Step #0 - "validate platform": - private-gke-standard-cluster.gke in .terraform/modules/private-gke-standard-cluster.gke/modules/private-cluster
Step #0 - "validate platform": - public-gke-autopilot-cluster in ../modules/gke-autopilot-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for public-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - public-gke-autopilot-cluster.gke in .terraform/modules/public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #0 - "validate platform": - public-gke-standard-cluster in ../modules/gke-standard-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for public-gke-standard-cluster.gke...
Step #0 - "validate platform": - public-gke-standard-cluster.gke in .terraform/modules/public-gke-standard-cluster.gke
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing provider plugins...
Step #0 - "validate platform": - Finding hashicorp/google versions matching ">= 3.33.0, >= 3.83.0, >= 4.25.0, >= 4.51.0, >= 4.64.0, >= 5.0.0, < 6.0.0"...
Step #0 - "validate platform": - Finding hashicorp/google-beta versions matching ">= 4.64.0, >= 5.0.0, < 6.0.0"...
Step #0 - "validate platform": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #0 - "validate platform": - Finding hashicorp/kubernetes versions matching "~> 2.10, 2.18.1"...
Step #0 - "validate platform": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #0 - "validate platform": - Installing hashicorp/helm v2.8.0...
Step #0 - "validate platform": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/kubernetes v2.18.1...
Step #0 - "validate platform": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/random v3.6.0...
Step #0 - "validate platform": - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/google v5.19.0...
Step #0 - "validate platform": - Installed hashicorp/google v5.19.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/google-beta v5.19.0...
Step #0 - "validate platform": - Installed hashicorp/google-beta v5.19.0 (signed by HashiCorp)
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #0 - "validate platform": selections it made above. Include this file in your version control repository
Step #0 - "validate platform": so that Terraform can guarantee to make the same selections by default when
Step #0 - "validate platform": you run "terraform init" in the future.
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has been successfully initialized!
Step #0 - "validate platform": 
Step #0 - "validate platform": You may now begin working with Terraform. Try running "terraform plan" to see
Step #0 - "validate platform": any changes that are required for your infrastructure. All Terraform commands
Step #0 - "validate platform": should now work.
Step #0 - "validate platform": 
Step #0 - "validate platform": If you ever set or change modules or backend configuration for Terraform,
Step #0 - "validate platform": rerun this command to reinitialize your working directory. If you forget, other
Step #0 - "validate platform": commands will detect it and remind you to do so if necessary.
Step #0 - "validate platform": Success! The configuration is valid.
Step #0 - "validate platform": 
Finished Step #0 - "validate platform"
Starting Step #2 - "validate jupyterhub"
Starting Step #1 - "validate ray"
Starting Step #3 - "validate rag"
Step #1 - "validate ray": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #2 - "validate jupyterhub": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #3 - "validate rag": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing the backend...
Step #1 - "validate ray": Initializing modules...
Step #1 - "validate ray": - gcs in ../../modules/gcs
Step #1 - "validate ray": - infra in ../../infrastructure
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing the backend...
Step #3 - "validate rag": Initializing modules...
Step #3 - "validate rag": - cloudsql in ../../modules/cloudsql
Step #3 - "validate rag": - frontend in frontend
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for frontend.frontend-workload-identity...
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing the backend...
Step #2 - "validate jupyterhub": Initializing modules...
Step #2 - "validate jupyterhub": - gcs in ../../modules/gcs
Step #2 - "validate jupyterhub": - infra in ../../infrastructure
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #1 - "validate ray": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/network/google 8.0.0 for infra.custom-network...
Step #2 - "validate jupyterhub": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/network/google 8.0.0 for infra.custom-network...
Step #1 - "validate ray": - infra.custom-network in .terraform/modules/infra.custom-network
Step #1 - "validate ray": - infra.custom-network.firewall_rules in .terraform/modules/infra.custom-network/modules/firewall-rules
Step #1 - "validate ray": - infra.custom-network.routes in .terraform/modules/infra.custom-network/modules/routes
Step #1 - "validate ray": - infra.custom-network.subnets in .terraform/modules/infra.custom-network/modules/subnets
Step #1 - "validate ray": - infra.custom-network.vpc in .terraform/modules/infra.custom-network/modules/vpc
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - frontend.frontend-workload-identity in .terraform/modules/frontend.frontend-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for frontend.frontend-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - infra.custom-network in .terraform/modules/infra.custom-network
Step #2 - "validate jupyterhub": - infra.custom-network.firewall_rules in .terraform/modules/infra.custom-network/modules/firewall-rules
Step #2 - "validate jupyterhub": - infra.custom-network.routes in .terraform/modules/infra.custom-network/modules/routes
Step #2 - "validate jupyterhub": - infra.custom-network.subnets in .terraform/modules/infra.custom-network/modules/subnets
Step #2 - "validate jupyterhub": - infra.custom-network.vpc in .terraform/modules/infra.custom-network/modules/vpc
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa in .terraform/modules/frontend.frontend-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/frontend.frontend-workload-identity.annotate-sa
Step #3 - "validate rag": - frontend.iap_auth in ../../modules/iap
Step #3 - "validate rag": - gcs in ../../modules/gcs
Step #3 - "validate rag": - inference-server in ../../tutorials/hf-tgi
Step #3 - "validate rag": - infra in ../../infrastructure
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #1 - "validate ray": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #1 - "validate ray": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #1 - "validate ray": - kuberay-cluster in ../../modules/kuberay-cluster
Step #1 - "validate ray": - kuberay-logging in ../../modules/kuberay-logging
Step #1 - "validate ray": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #3 - "validate rag": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #1 - "validate ray": - kuberay-operator in ../../modules/kuberay-operator
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-operator.kuberay-workload-identity...
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/network/google 8.0.0 for infra.custom-network...
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #2 - "validate jupyterhub": - jupyterhub in ../../modules/jupyter
Step #2 - "validate jupyterhub": - jupyterhub.iap_auth in ../../modules/iap
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #3 - "validate rag": - infra.custom-network in .terraform/modules/infra.custom-network
Step #3 - "validate rag": - infra.custom-network.firewall_rules in .terraform/modules/infra.custom-network/modules/firewall-rules
Step #3 - "validate rag": - infra.custom-network.routes in .terraform/modules/infra.custom-network/modules/routes
Step #3 - "validate rag": - infra.custom-network.subnets in .terraform/modules/infra.custom-network/modules/subnets
Step #3 - "validate rag": - infra.custom-network.vpc in .terraform/modules/infra.custom-network/modules/vpc
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - kuberay-operator.kuberay-workload-identity in .terraform/modules/kuberay-operator.kuberay-workload-identity/modules/workload-identity
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for kuberay-operator.kuberay-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #1 - "validate ray": - kuberay-operator.kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-operator.kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #1 - "validate ray": - kuberay-operator.kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-operator.kuberay-workload-identity.annotate-sa
Step #1 - "validate ray": - namespace in ../../modules/kubernetes-namespace
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing provider plugins...
Step #1 - "validate ray": - Finding hashicorp/google versions matching ">= 3.33.0, >= 3.39.0, >= 3.53.0, >= 3.83.0, >= 4.25.0, >= 4.51.0, >= 4.64.0, >= 5.0.0, < 6.0.0"...
Step #1 - "validate ray": - Finding hashicorp/google-beta versions matching ">= 4.64.0, >= 5.0.0, < 6.0.0"...
Step #1 - "validate ray": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #3 - "validate rag": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.private-gke-standard-cluster.gke...
Step #3 - "validate rag": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #3 - "validate rag": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 29.0.0 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #3 - "validate rag": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #3 - "validate rag": - jupyterhub in ../../modules/jupyter
Step #3 - "validate rag": - jupyterhub.iap_auth in ../../modules/iap
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #1 - "validate ray": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #3 - "validate rag": - kuberay-cluster in ../../modules/kuberay-cluster
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - kuberay-logging in ../../modules/kuberay-logging
Step #3 - "validate rag": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #3 - "validate rag": - kuberay-operator in ../../modules/kuberay-operator
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-operator.kuberay-workload-identity...
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #2 - "validate jupyterhub": - namespace in ../../modules/kubernetes-namespace
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing provider plugins...
Step #2 - "validate jupyterhub": - Finding hashicorp/google-beta versions matching ">= 4.64.0, >= 5.0.0, < 6.0.0"...
Step #1 - "validate ray": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #3 - "validate rag": - kuberay-operator.kuberay-workload-identity in .terraform/modules/kuberay-operator.kuberay-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.0 for kuberay-operator.kuberay-workload-identity.annotate-sa...
Step #3 - "validate rag": - kuberay-operator.kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-operator.kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - kuberay-operator.kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-operator.kuberay-workload-identity.annotate-sa
Step #3 - "validate rag": - namespace in ../../modules/kubernetes-namespace
Step #2 - "validate jupyterhub": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing provider plugins...
Step #3 - "validate rag": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #3 - "validate rag": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #2 - "validate jupyterhub": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #3 - "validate rag": - Finding hashicorp/google versions matching ">= 3.33.0, >= 3.39.0, >= 3.53.0, >= 3.83.0, >= 4.25.0, >= 4.51.0, >= 4.64.0, >= 5.0.0, < 6.0.0"...
Step #1 - "validate ray": - Installing hashicorp/google-beta v5.19.0...
Step #2 - "validate jupyterhub": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #3 - "validate rag": - Finding hashicorp/google-beta versions matching ">= 4.64.0, >= 5.0.0, < 6.0.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #3 - "validate rag": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/google versions matching ">= 3.33.0, >= 3.39.0, >= 3.53.0, >= 3.83.0, >= 4.25.0, >= 4.51.0, >= 4.64.0, >= 5.0.0, < 6.0.0"...
Step #3 - "validate rag": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #3 - "validate rag": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #2 - "validate jupyterhub": - Installing hashicorp/google v5.19.0...
Step #3 - "validate rag": - Installing hashicorp/google v5.19.0...
Step #1 - "validate ray": - Installed hashicorp/google-beta v5.19.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/helm v2.8.0...
Step #2 - "validate jupyterhub": - Installed hashicorp/google v5.19.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installed hashicorp/google v5.19.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/google-beta v5.19.0...
Step #3 - "validate rag": - Installing hashicorp/google-beta v5.19.0...
Step #1 - "validate ray": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/kubernetes v2.18.1...
Step #1 - "validate ray": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/google-beta v5.19.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installed hashicorp/google-beta v5.19.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/null v3.2.2...
Step #2 - "validate jupyterhub": - Installing hashicorp/random v3.6.0...
Step #3 - "validate rag": - Installing hashicorp/helm v2.8.0...
Step #1 - "validate ray": - Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/external v2.3.3...
Step #2 - "validate jupyterhub": - Installing hashicorp/external v2.3.3...
Step #1 - "validate ray": - Installed hashicorp/external v2.3.3 (signed by HashiCorp)
Step #3 - "validate rag": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/external v2.3.3 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/random v3.6.0...
Step #3 - "validate rag": - Installing hashicorp/kubernetes v2.18.1...
Step #2 - "validate jupyterhub": - Installing hashicorp/null v3.2.2...
Step #1 - "validate ray": - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/google v5.19.0...
Step #2 - "validate jupyterhub": - Installing hashicorp/helm v2.8.0...
Step #3 - "validate rag": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/random v3.6.0...
Step #3 - "validate rag": - Installed hashicorp/random v3.6.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/null v3.2.2...
Step #2 - "validate jupyterhub": - Installing hashicorp/kubernetes v2.18.1...
Step #3 - "validate rag": - Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/google v5
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/1a17ac6c-140e-4f89-b838-453b2f3e1349?project=142966328212.]
...
r":               - "gke-ml-f412967-281-cluster-gpu-pool-t4",
Step #11 - "cleanup gke cluster":               - "gke-node",
Step #11 - "cleanup gke cluster":               - "ai-on-gke",
Step #11 - "cleanup gke cluster":             ] -> null
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - gcfs_config {
Step #11 - "cleanup gke cluster":               - enabled = true -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - shielded_instance_config {
Step #11 - "cleanup gke cluster":               - enable_integrity_monitoring = true -> null
Step #11 - "cleanup gke cluster":               - enable_secure_boot          = false -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - workload_metadata_config {
Step #11 - "cleanup gke cluster":               - mode = "GKE_METADATA" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - timeouts {
Step #11 - "cleanup gke cluster":           - create = "45m" -> null
Step #11 - "cleanup gke cluster":           - delete = "45m" -> null
Step #11 - "cleanup gke cluster":           - update = "45m" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - upgrade_settings {
Step #11 - "cleanup gke cluster":           - max_surge       = 1 -> null
Step #11 - "cleanup gke cluster":           - max_unavailable = 0 -> null
Step #11 - "cleanup gke cluster":           - strategy        = "SURGE" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-nodeService_account" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYTKvB9aSg=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/container.nodeServiceAccount/serviceAccount:tf-gke-ml-f412967-281--2krs@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-f412967-281--2krs@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/container.nodeServiceAccount" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.google_service_account.cluster_service_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_service_account" "cluster_service_account" {
Step #11 - "cleanup gke cluster":       - account_id   = "tf-gke-ml-f412967-281--2krs" -> null
Step #11 - "cleanup gke cluster":       - disabled     = false -> null
Step #11 - "cleanup gke cluster":       - display_name = "Terraform-managed service account for cluster ml-f412967-281-cluster" -> null
Step #11 - "cleanup gke cluster":       - email        = "tf-gke-ml-f412967-281--2krs@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - id           = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-f412967-281--2krs@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member       = "serviceAccount:tf-gke-ml-f412967-281--2krs@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - name         = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-f412967-281--2krs@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project      = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - unique_id    = "105402819558995900936" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.random_shuffle.available_zones[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "random_shuffle" "available_zones" {
Step #11 - "cleanup gke cluster":       - id           = "-" -> null
Step #11 - "cleanup gke cluster":       - input        = [
Step #11 - "cleanup gke cluster":           - "us-central1-a",
Step #11 - "cleanup gke cluster":           - "us-central1-b",
Step #11 - "cleanup gke cluster":           - "us-central1-c",
Step #11 - "cleanup gke cluster":           - "us-central1-f",
Step #11 - "cleanup gke cluster":         ] -> null
Step #11 - "cleanup gke cluster":       - result       = [
Step #11 - "cleanup gke cluster":           - "us-central1-a",
Step #11 - "cleanup gke cluster":           - "us-central1-f",
Step #11 - "cleanup gke cluster":           - "us-central1-c",
Step #11 - "cleanup gke cluster":         ] -> null
Step #11 - "cleanup gke cluster":       - result_count = 3 -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.random_string.cluster_service_account_suffix will be destroyed
Step #11 - "cleanup gke cluster":   - resource "random_string" "cluster_service_account_suffix" {
Step #11 - "cleanup gke cluster":       - id          = "2krs" -> null
Step #11 - "cleanup gke cluster":       - length      = 4 -> null
Step #11 - "cleanup gke cluster":       - lower       = true -> null
Step #11 - "cleanup gke cluster":       - min_lower   = 0 -> null
Step #11 - "cleanup gke cluster":       - min_numeric = 0 -> null
Step #11 - "cleanup gke cluster":       - min_special = 0 -> null
Step #11 - "cleanup gke cluster":       - min_upper   = 0 -> null
Step #11 - "cleanup gke cluster":       - number      = true -> null
Step #11 - "cleanup gke cluster":       - numeric     = true -> null
Step #11 - "cleanup gke cluster":       - result      = "2krs" -> null
Step #11 - "cleanup gke cluster":       - special     = false -> null
Step #11 - "cleanup gke cluster":       - upper       = false -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Plan: 0 to add, 0 to change, 8 to destroy.
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Changes to Outputs:
Step #11 - "cleanup gke cluster":   - ca_certificate   = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - cluster_location = "us-central1" -> null
Step #11 - "cleanup gke cluster":   - cluster_name     = "ml-f412967-281-cluster" -> null
Step #11 - "cleanup gke cluster":   - endpoint         = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - private_cluster  = false -> null
Step #11 - "cleanup gke cluster":   - project_id       = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destroying... [id=gke-ai-eco-dev/roles/container.nodeServiceAccount/serviceAccount:tf-gke-ml-f412967-281--2krs@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster/nodePools/cpu-pool]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster/nodePools/gpu-pool-l4]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster/nodePools/gpu-pool-t4]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destruction complete after 7s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-l4, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-l4, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-l4, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-l4, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-l4, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-l4, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-l4, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-l4"]: Destruction complete after 1m10s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...2967-281-cluster/nodePools/gpu-pool-t4, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool-t4"]: Destruction complete after 1m41s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 1m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 2m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 2m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 2m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 2m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 2m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 2m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 3m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 3m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 3m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 3m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 3m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 3m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 4m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 4m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 4m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ce...f412967-281-cluster/nodePools/cpu-pool, 4m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Destruction complete after 4m31s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 1m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 2m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 2m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 2m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 2m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 2m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 2m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 3m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 3m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 3m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 3m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 3m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 3m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 4m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 4m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 4m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 4m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 4m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 4m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 5m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 5m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 5m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 5m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 5m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 5m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 6m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 6m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 6m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 6m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-central1/clusters/ml-f412967-281-cluster, 6m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Destruction complete after 6m48s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destroying... [id=projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-f412967-281--2krs@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.random_shuffle.available_zones[0]: Destroying... [id=-]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.random_shuffle.available_zones[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.random_string.cluster_service_account_suffix: Destroying... [id=2krs]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.random_string.cluster_service_account_suffix: Destruction complete after 0s
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Destroy complete! Resources: 8 destroyed.
Finished Step #11 - "cleanup gke cluster"
Starting Step #12 - "check result"
Step #12 - "check result": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Finished Step #12 - "check result"
PUSH
DONE

Build Log: https://console.cloud.google.com/cloud-build/builds;region=us-central1/1a17ac6c-140e-4f89-b838-453b2f3e1349?project=142966328212