Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIX] Single GPU training job update #794

Merged

single GPU job update

8f4839d
Select commit
Loading
Failed to load commit list.
Merged

[FIX] Single GPU training job update #794

single GPU job update
8f4839d
Select commit
Loading
Failed to load commit list.
Google Cloud Build / pr-review-trigger (gke-ai-eco-dev) succeeded Sep 3, 2024 in 53m 35s

Summary

Build Information

Trigger pr-review-trigger
Build 3a9541ca-2847-45c2-ad0b-b9f5acf4d5bb
Start 2024-09-03T08:30:47-07:00
Duration 52m40.215s
Status SUCCESS

Steps

Step Status Duration
validate platform SUCCESS 1m0.088s
validate ray SUCCESS 19.715s
validate jupyterhub SUCCESS 19.717s
validate rag SUCCESS 20.919s
create gke cluster SUCCESS 9m38.539s
test ray cluster SUCCESS 6m14.672s
cleanup ray cluster SUCCESS 33.548s
test jupyterhub SUCCESS 1m26.963s
cleanup jupyterhub SUCCESS 1m15.18s
test rag SUCCESS 30m0.62s
cleanup rag SUCCESS 2m20.066s
cleanup gke cluster FAILURE 9m15.127s
check result SUCCESS 759ms

Details

starting build "3a9541ca-2847-45c2-ad0b-b9f5acf4d5bb"

FETCHSOURCE
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /workspace/.git/
From https://github.com/GoogleCloudPlatform/ai-on-gke
 * branch            8f4839dd2bbaab34489733af394cee2a9831bcf1 -> FETCH_HEAD
HEAD is now at 8f4839d single GPU job update
SETUPBUILD
BUILD
Starting Step #0 - "validate platform"
Step #0 - "validate platform": Pulling image: gcr.io/gke-ai-eco-dev/terraform
Step #0 - "validate platform": Using default tag: latest
Step #0 - "validate platform": latest: Pulling from gke-ai-eco-dev/terraform
Step #0 - "validate platform": ddf874abf16c: Pulling fs layer
Step #0 - "validate platform": ecfb911cf53f: Pulling fs layer
Step #0 - "validate platform": ce689bb70bd5: Pulling fs layer
Step #0 - "validate platform": 7d2b54998739: Pulling fs layer
Step #0 - "validate platform": b1c39b09da0f: Pulling fs layer
Step #0 - "validate platform": cc032834b94b: Pulling fs layer
Step #0 - "validate platform": 0279610fca41: Pulling fs layer
Step #0 - "validate platform": 6f125b563dda: Pulling fs layer
Step #0 - "validate platform": 7c8b6e22fcff: Pulling fs layer
Step #0 - "validate platform": faaaa1840355: Pulling fs layer
Step #0 - "validate platform": 60a28c2199b1: Pulling fs layer
Step #0 - "validate platform": 4c7171ac56db: Pulling fs layer
Step #0 - "validate platform": 372d85ec2f1f: Pulling fs layer
Step #0 - "validate platform": 633b411e5bd5: Pulling fs layer
Step #0 - "validate platform": 60a28c2199b1: Waiting
Step #0 - "validate platform": 4c7171ac56db: Waiting
Step #0 - "validate platform": 372d85ec2f1f: Waiting
Step #0 - "validate platform": 633b411e5bd5: Waiting
Step #0 - "validate platform": 7d2b54998739: Waiting
Step #0 - "validate platform": b1c39b09da0f: Waiting
Step #0 - "validate platform": cc032834b94b: Waiting
Step #0 - "validate platform": 0279610fca41: Waiting
Step #0 - "validate platform": 7c8b6e22fcff: Waiting
Step #0 - "validate platform": 6f125b563dda: Waiting
Step #0 - "validate platform": ecfb911cf53f: Verifying Checksum
Step #0 - "validate platform": ecfb911cf53f: Download complete
Step #0 - "validate platform": 7d2b54998739: Download complete
Step #0 - "validate platform": ce689bb70bd5: Verifying Checksum
Step #0 - "validate platform": ce689bb70bd5: Download complete
Step #0 - "validate platform": ddf874abf16c: Verifying Checksum
Step #0 - "validate platform": ddf874abf16c: Download complete
Step #0 - "validate platform": 0279610fca41: Verifying Checksum
Step #0 - "validate platform": 0279610fca41: Download complete
Step #0 - "validate platform": 6f125b563dda: Verifying Checksum
Step #0 - "validate platform": 6f125b563dda: Download complete
Step #0 - "validate platform": 7c8b6e22fcff: Verifying Checksum
Step #0 - "validate platform": 7c8b6e22fcff: Download complete
Step #0 - "validate platform": faaaa1840355: Verifying Checksum
Step #0 - "validate platform": faaaa1840355: Download complete
Step #0 - "validate platform": cc032834b94b: Verifying Checksum
Step #0 - "validate platform": cc032834b94b: Download complete
Step #0 - "validate platform": 60a28c2199b1: Verifying Checksum
Step #0 - "validate platform": 60a28c2199b1: Download complete
Step #0 - "validate platform": 372d85ec2f1f: Download complete
Step #0 - "validate platform": 4c7171ac56db: Verifying Checksum
Step #0 - "validate platform": 4c7171ac56db: Download complete
Step #0 - "validate platform": ddf874abf16c: Pull complete
Step #0 - "validate platform": ecfb911cf53f: Pull complete
Step #0 - "validate platform": ce689bb70bd5: Pull complete
Step #0 - "validate platform": 7d2b54998739: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Verifying Checksum
Step #0 - "validate platform": 633b411e5bd5: Download complete
Step #0 - "validate platform": b1c39b09da0f: Verifying Checksum
Step #0 - "validate platform": b1c39b09da0f: Download complete
Step #0 - "validate platform": b1c39b09da0f: Pull complete
Step #0 - "validate platform": cc032834b94b: Pull complete
Step #0 - "validate platform": 0279610fca41: Pull complete
Step #0 - "validate platform": 6f125b563dda: Pull complete
Step #0 - "validate platform": 7c8b6e22fcff: Pull complete
Step #0 - "validate platform": faaaa1840355: Pull complete
Step #0 - "validate platform": 60a28c2199b1: Pull complete
Step #0 - "validate platform": 4c7171ac56db: Pull complete
Step #0 - "validate platform": 372d85ec2f1f: Pull complete
Step #0 - "validate platform": 633b411e5bd5: Pull complete
Step #0 - "validate platform": Digest: sha256:202db292b551d530ff51210759dd25971cb64b5afe675a5616e0c244d48a452b
Step #0 - "validate platform": Status: Downloaded newer image for gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": gcr.io/gke-ai-eco-dev/terraform:latest
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing the backend...
Step #0 - "validate platform": Initializing modules...
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for cloud-nat...
Step #0 - "validate platform": - cloud-nat in .terraform/modules/cloud-nat
Step #0 - "validate platform": - custom-network in ../modules/gcp-network
Step #0 - "validate platform": - private-gke-autopilot-cluster in ../modules/gke-autopilot-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for private-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - private-gke-autopilot-cluster.gke in .terraform/modules/private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #0 - "validate platform": - private-gke-standard-cluster in ../modules/gke-standard-private-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for private-gke-standard-cluster.gke...
Step #0 - "validate platform": - private-gke-standard-cluster.gke in .terraform/modules/private-gke-standard-cluster.gke/modules/private-cluster
Step #0 - "validate platform": - public-gke-autopilot-cluster in ../modules/gke-autopilot-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for public-gke-autopilot-cluster.gke...
Step #0 - "validate platform": - public-gke-autopilot-cluster.gke in .terraform/modules/public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #0 - "validate platform": - public-gke-standard-cluster in ../modules/gke-standard-public-cluster
Step #0 - "validate platform": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for public-gke-standard-cluster.gke...
Step #0 - "validate platform": - public-gke-standard-cluster.gke in .terraform/modules/public-gke-standard-cluster.gke
Step #0 - "validate platform": 
Step #0 - "validate platform": Initializing provider plugins...
Step #0 - "validate platform": - Finding hashicorp/kubernetes versions matching "~> 2.10, 2.18.1"...
Step #0 - "validate platform": - Finding hashicorp/google versions matching ">= 4.51.0, >= 5.40.0, < 6.0.0"...
Step #0 - "validate platform": - Finding hashicorp/google-beta versions matching ">= 5.40.0, < 6.0.0"...
Step #0 - "validate platform": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #0 - "validate platform": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #0 - "validate platform": - Installing hashicorp/google-beta v5.43.1...
Step #0 - "validate platform": - Installed hashicorp/google-beta v5.43.1 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/helm v2.8.0...
Step #0 - "validate platform": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/random v3.6.2...
Step #0 - "validate platform": - Installed hashicorp/random v3.6.2 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/kubernetes v2.18.1...
Step #0 - "validate platform": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #0 - "validate platform": - Installing hashicorp/google v5.43.1...
Step #0 - "validate platform": - Installed hashicorp/google v5.43.1 (signed by HashiCorp)
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #0 - "validate platform": selections it made above. Include this file in your version control repository
Step #0 - "validate platform": so that Terraform can guarantee to make the same selections by default when
Step #0 - "validate platform": you run "terraform init" in the future.
Step #0 - "validate platform": 
Step #0 - "validate platform": Terraform has been successfully initialized!
Step #0 - "validate platform": 
Step #0 - "validate platform": You may now begin working with Terraform. Try running "terraform plan" to see
Step #0 - "validate platform": any changes that are required for your infrastructure. All Terraform commands
Step #0 - "validate platform": should now work.
Step #0 - "validate platform": 
Step #0 - "validate platform": If you ever set or change modules or backend configuration for Terraform,
Step #0 - "validate platform": rerun this command to reinitialize your working directory. If you forget, other
Step #0 - "validate platform": commands will detect it and remind you to do so if necessary.
Step #0 - "validate platform": Success! The configuration is valid.
Step #0 - "validate platform": 
Finished Step #0 - "validate platform"
Starting Step #2 - "validate jupyterhub"
Starting Step #3 - "validate rag"
Starting Step #1 - "validate ray"
Step #2 - "validate jupyterhub": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #1 - "validate ray": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #3 - "validate rag": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing the backend...
Step #3 - "validate rag": Initializing modules...
Step #3 - "validate rag": - cloudsql in ../../modules/cloudsql
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/sql-db/google 20.0.0 for cloudsql.cloudsql...
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing the backend...
Step #1 - "validate ray": Initializing modules...
Step #1 - "validate ray": - gcs in ../../modules/gcs
Step #1 - "validate ray": - infra in ../../infrastructure
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing the backend...
Step #2 - "validate jupyterhub": Initializing modules...
Step #2 - "validate jupyterhub": - gcs in ../../modules/gcs
Step #2 - "validate jupyterhub": - infra in ../../infrastructure
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #1 - "validate ray": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #1 - "validate ray": - infra.custom-network in ../../modules/gcp-network
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #2 - "validate jupyterhub": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #2 - "validate jupyterhub": - infra.custom-network in ../../modules/gcp-network
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.private-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.private-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - cloudsql.cloudsql in .terraform/modules/cloudsql.cloudsql/modules/postgresql
Step #3 - "validate rag": - frontend in frontend
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for frontend.frontend-workload-identity...
Step #1 - "validate ray": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #1 - "validate ray": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.private-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.private-gke-standard-cluster.gke...
Step #1 - "validate ray": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.public-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #1 - "validate ray": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.public-gke-standard-cluster.gke...
Step #2 - "validate jupyterhub": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.public-gke-autopilot-cluster.gke...
Step #3 - "validate rag": - frontend.frontend-workload-identity in .terraform/modules/frontend.frontend-workload-identity/modules/workload-identity
Step #1 - "validate ray": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #2 - "validate jupyterhub": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - kuberay-cluster in ../../modules/kuberay-cluster
Step #1 - "validate ray": - kuberay-cluster.iap_auth in ../../modules/iap
Step #1 - "validate ray": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-workload-identity...
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.1 for frontend.frontend-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #2 - "validate jupyterhub": - jupyterhub in ../../modules/jupyter
Step #2 - "validate jupyterhub": - jupyterhub.iap_auth in ../../modules/iap
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa in .terraform/modules/frontend.frontend-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - frontend.frontend-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/frontend.frontend-workload-identity.annotate-sa
Step #3 - "validate rag": - frontend.iap_auth in ../../modules/iap
Step #3 - "validate rag": - gcs in ../../modules/gcs
Step #3 - "validate rag": - inference-server in ../../modules/inference-service
Step #3 - "validate rag": - infra in ../../infrastructure
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/cloud-nat/google 5.0.0 for infra.cloud-nat...
Step #3 - "validate rag": - infra.cloud-nat in .terraform/modules/infra.cloud-nat
Step #3 - "validate rag": - infra.custom-network in ../../modules/gcp-network
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster in ../../modules/gke-autopilot-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.private-gke-autopilot-cluster.gke...
Step #1 - "validate ray": - kuberay-workload-identity in .terraform/modules/kuberay-workload-identity/modules/workload-identity
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.1 for kuberay-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.1 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #1 - "validate ray": - kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #1 - "validate ray": - kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-workload-identity.annotate-sa
Step #1 - "validate ray": - namespace in ../../modules/kubernetes-namespace
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #2 - "validate jupyterhub": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #2 - "validate jupyterhub": - namespace in ../../modules/kubernetes-namespace
Step #1 - "validate ray": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #2 - "validate jupyterhub": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #3 - "validate rag": - infra.private-gke-autopilot-cluster.gke in .terraform/modules/infra.private-gke-autopilot-cluster.gke/modules/beta-autopilot-private-cluster
Step #3 - "validate rag": - infra.private-gke-standard-cluster in ../../modules/gke-standard-private-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.private-gke-standard-cluster.gke...
Step #1 - "validate ray": - project-services in .terraform/modules/project-services/modules/project_services
Step #1 - "validate ray": 
Step #1 - "validate ray": Initializing provider plugins...
Step #1 - "validate ray": - terraform.io/builtin/terraform is built in to Terraform
Step #1 - "validate ray": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.40.0, < 6.0.0"...
Step #1 - "validate ray": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #3 - "validate rag": - infra.private-gke-standard-cluster.gke in .terraform/modules/infra.private-gke-standard-cluster.gke/modules/private-cluster
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster in ../../modules/gke-autopilot-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.public-gke-autopilot-cluster.gke...
Step #2 - "validate jupyterhub": - project-services in .terraform/modules/project-services/modules/project_services
Step #1 - "validate ray": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Initializing provider plugins...
Step #2 - "validate jupyterhub": - terraform.io/builtin/terraform is built in to Terraform
Step #2 - "validate jupyterhub": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.40.0, < 6.0.0"...
Step #3 - "validate rag": - infra.public-gke-autopilot-cluster.gke in .terraform/modules/infra.public-gke-autopilot-cluster.gke/modules/beta-autopilot-public-cluster
Step #3 - "validate rag": - infra.public-gke-standard-cluster in ../../modules/gke-standard-public-cluster
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 32.0.1 for infra.public-gke-standard-cluster.gke...
Step #1 - "validate ray": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #1 - "validate ray": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #2 - "validate jupyterhub": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0"...
Step #3 - "validate rag": - infra.public-gke-standard-cluster.gke in .terraform/modules/infra.public-gke-standard-cluster.gke
Step #3 - "validate rag": - jupyterhub in ../../modules/jupyter
Step #3 - "validate rag": - jupyterhub.iap_auth in ../../modules/iap
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for jupyterhub.jupyterhub-workload-identity...
Step #1 - "validate ray": - Finding hashicorp/time versions matching "0.11.1"...
Step #2 - "validate jupyterhub": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #1 - "validate ray": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.40.0, < 6.0.0"...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity in .terraform/modules/jupyterhub.jupyterhub-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.1 for jupyterhub.jupyterhub-workload-identity.annotate-sa...
Step #2 - "validate jupyterhub": - Finding hashicorp/null versions matching ">= 2.1.0"...
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - jupyterhub.jupyterhub-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/jupyterhub.jupyterhub-workload-identity.annotate-sa
Step #3 - "validate rag": - kuberay-cluster in ../../modules/kuberay-cluster
Step #3 - "validate rag": - kuberay-cluster.iap_auth in ../../modules/iap
Step #3 - "validate rag": - kuberay-monitoring in ../../modules/kuberay-monitoring
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/kubernetes-engine/google 30.0.0 for kuberay-workload-identity...
Step #1 - "validate ray": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #2 - "validate jupyterhub": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #3 - "validate rag": - kuberay-workload-identity in .terraform/modules/kuberay-workload-identity/modules/workload-identity
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/gcloud/google 3.4.1 for kuberay-workload-identity.annotate-sa...
Step #3 - "validate rag": - kuberay-workload-identity.annotate-sa in .terraform/modules/kuberay-workload-identity.annotate-sa/modules/kubectl-wrapper
Step #3 - "validate rag": - kuberay-workload-identity.annotate-sa.gcloud_kubectl in .terraform/modules/kuberay-workload-identity.annotate-sa
Step #3 - "validate rag": - namespace in ../../modules/kubernetes-namespace
Step #2 - "validate jupyterhub": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.40.0, < 6.0.0"...
Step #3 - "validate rag": Downloading registry.terraform.io/terraform-google-modules/project-factory/google 14.5.0 for project-services...
Step #1 - "validate ray": - Installing hashicorp/google-beta v5.43.1...
Step #2 - "validate jupyterhub": - Installing hashicorp/external v2.3.3...
Step #2 - "validate jupyterhub": - Installed hashicorp/external v2.3.3 (signed by HashiCorp)
Step #3 - "validate rag": - project-services in .terraform/modules/project-services/modules/project_services
Step #2 - "validate jupyterhub": - Installing hashicorp/null v3.2.2...
Step #3 - "validate rag": 
Step #3 - "validate rag": Initializing provider plugins...
Step #3 - "validate rag": - terraform.io/builtin/terraform is built in to Terraform
Step #3 - "validate rag": - Finding hashicorp/time versions matching "0.11.1"...
Step #3 - "validate rag": - Finding hashicorp/helm versions matching "~> 2.8.0"...
Step #3 - "validate rag": - Finding hashicorp/random versions matching ">= 2.1.0, ~> 3.0, ~> 3.1"...
Step #3 - "validate rag": - Finding hashicorp/google versions matching ">= 3.39.0, >= 3.43.0, >= 3.53.0, >= 4.51.0, >= 5.12.0, >= 5.40.0, < 6.0.0"...
Step #3 - "validate rag": - Finding hashicorp/google-beta versions matching ">= 3.43.0, >= 5.12.0, >= 5.40.0, < 6.0.0"...
Step #3 - "validate rag": - Finding hashicorp/kubernetes versions matching "~> 2.10, ~> 2.13, 2.18.1"...
Step #3 - "validate rag": - Finding hashicorp/external versions matching ">= 2.2.2"...
Step #3 - "validate rag": - Finding hashicorp/null versions matching ">= 2.1.0, ~> 3.1"...
Step #2 - "validate jupyterhub": - Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/kubernetes v2.18.1...
Step #3 - "validate rag": - Installing hashicorp/helm v2.8.0...
Step #1 - "validate ray": - Installed hashicorp/google-beta v5.43.1 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/helm v2.8.0...
Step #2 - "validate jupyterhub": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/google v5.43.1...
Step #3 - "validate rag": - Installing hashicorp/random v3.6.2...
Step #3 - "validate rag": - Installed hashicorp/random v3.6.2 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/google v5.43.1...
Step #1 - "validate ray": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/google v5.43.1...
Step #2 - "validate jupyterhub": - Installed hashicorp/google v5.43.1 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/google-beta v5.43.1...
Step #3 - "validate rag": - Installed hashicorp/google v5.43.1 (signed by HashiCorp)
Step #3 - "validate rag": - Installing hashicorp/google-beta v5.43.1...
Step #1 - "validate ray": - Installed hashicorp/google v5.43.1 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/random v3.6.2...
Step #1 - "validate ray": - Installed hashicorp/random v3.6.2 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/external v2.3.3...
Step #2 - "validate jupyterhub": - Installed hashicorp/google-beta v5.43.1 (signed by HashiCorp)
Step #1 - "validate ray": - Installed hashicorp/external v2.3.3 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/helm v2.8.0...
Step #1 - "validate ray": - Installing hashicorp/null v3.2.2...
Step #1 - "validate ray": - Installed hashicorp/null v3.2.2 (signed by HashiCorp)
Step #3 - "validate rag": - Installed hashicorp/google-beta v5.43.1 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/kubernetes v2.18.1...
Step #3 - "validate rag": - Installing hashicorp/kubernetes v2.18.1...
Step #2 - "validate jupyterhub": - Installed hashicorp/helm v2.8.0 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installing hashicorp/random v3.6.2...
Step #1 - "validate ray": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #2 - "validate jupyterhub": - Installed hashicorp/random v3.6.2 (signed by HashiCorp)
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #2 - "validate jupyterhub": selections it made above. Include this file in your version control repository
Step #2 - "validate jupyterhub": so that Terraform can guarantee to make the same selections by default when
Step #2 - "validate jupyterhub": you run "terraform init" in the future.
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": Terraform has been successfully initialized!
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": You may now begin working with Terraform. Try running "terraform plan" to see
Step #2 - "validate jupyterhub": any changes that are required for your infrastructure. All Terraform commands
Step #2 - "validate jupyterhub": should now work.
Step #2 - "validate jupyterhub": 
Step #2 - "validate jupyterhub": If you ever set or change modules or backend configuration for Terraform,
Step #2 - "validate jupyterhub": rerun this command to reinitialize your working directory. If you forget, other
Step #2 - "validate jupyterhub": commands will detect it and remind you to do so if necessary.
Step #3 - "validate rag": - Installed hashicorp/kubernetes v2.18.1 (signed by HashiCorp)
Step #1 - "validate ray": - Installing hashicorp/time v0.11.1...
Step #3 - "validate rag": - Installing hashicorp/external v2.3.3...
Step #1 - "validate ray": - Installed hashicorp/time v0.11.1 (signed by HashiCorp)
Step #1 - "validate ray": 
Step #1 - "validate ray": Terraform has created a lock file .terraform.lock.hcl to record the provider
Step #1 - "validate ray": selections it made above. Include this file in your version control repository
Step #1 - "validate ray": so that
...
[Logs truncated due to log size limitations. For full logs, see https://console.cloud.google.com/cloud-build/builds;region=us-central1/3a9541ca-2847-45c2-ad0b-b9f5acf4d5bb?project=142966328212.]
...
ster":               - enable_secure_boot          = false -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":           - workload_metadata_config {
Step #11 - "cleanup gke cluster":               - mode = "GKE_METADATA" -> null
Step #11 - "cleanup gke cluster":             }
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - timeouts {
Step #11 - "cleanup gke cluster":           - create = "45m" -> null
Step #11 - "cleanup gke cluster":           - delete = "45m" -> null
Step #11 - "cleanup gke cluster":           - update = "45m" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":       - upgrade_settings {
Step #11 - "cleanup gke cluster":           - max_surge       = 1 -> null
Step #11 - "cleanup gke cluster":           - max_unavailable = 0 -> null
Step #11 - "cleanup gke cluster":           - strategy        = "SURGE" -> null
Step #11 - "cleanup gke cluster":         }
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-metric_writer" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYhOVQaIKs=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/monitoring.metricWriter/serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/monitoring.metricWriter" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-nodeService_account" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYhOVQaIKs=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/container.defaultNodeServiceAccount/serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/container.defaultNodeServiceAccount" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_project_iam_member" "cluster_service_account-resourceMetadata-writer" {
Step #11 - "cleanup gke cluster":       - etag    = "BwYhOVQaIKs=" -> null
Step #11 - "cleanup gke cluster":       - id      = "gke-ai-eco-dev/roles/stackdriver.resourceMetadata.writer/serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member  = "serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - role    = "roles/stackdriver.resourceMetadata.writer" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.google_service_account.cluster_service_account[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "google_service_account" "cluster_service_account" {
Step #11 - "cleanup gke cluster":       - account_id   = "tf-gke-ml-8f4839d-794--ob8a" -> null
Step #11 - "cleanup gke cluster":       - disabled     = false -> null
Step #11 - "cleanup gke cluster":       - display_name = "Terraform-managed service account for cluster ml-8f4839d-794-3a9541ca-cluster" -> null
Step #11 - "cleanup gke cluster":       - email        = "tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - id           = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - member       = "serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - name         = "projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com" -> null
Step #11 - "cleanup gke cluster":       - project      = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster":       - unique_id    = "105171510548468574198" -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster":   # module.public-gke-standard-cluster[0].module.gke.random_string.cluster_service_account_suffix[0] will be destroyed
Step #11 - "cleanup gke cluster":   - resource "random_string" "cluster_service_account_suffix" {
Step #11 - "cleanup gke cluster":       - id          = "ob8a" -> null
Step #11 - "cleanup gke cluster":       - length      = 4 -> null
Step #11 - "cleanup gke cluster":       - lower       = true -> null
Step #11 - "cleanup gke cluster":       - min_lower   = 0 -> null
Step #11 - "cleanup gke cluster":       - min_numeric = 0 -> null
Step #11 - "cleanup gke cluster":       - min_special = 0 -> null
Step #11 - "cleanup gke cluster":       - min_upper   = 0 -> null
Step #11 - "cleanup gke cluster":       - number      = true -> null
Step #11 - "cleanup gke cluster":       - numeric     = true -> null
Step #11 - "cleanup gke cluster":       - result      = "ob8a" -> null
Step #11 - "cleanup gke cluster":       - special     = false -> null
Step #11 - "cleanup gke cluster":       - upper       = false -> null
Step #11 - "cleanup gke cluster":     }
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Plan: 0 to add, 0 to change, 12 to destroy.
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Changes to Outputs:
Step #11 - "cleanup gke cluster":   - ca_certificate   = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - cluster_location = "us-east4" -> null
Step #11 - "cleanup gke cluster":   - cluster_name     = "ml-8f4839d-794-3a9541ca-cluster" -> null
Step #11 - "cleanup gke cluster":   - endpoint         = (sensitive value) -> null
Step #11 - "cleanup gke cluster":   - private_cluster  = false -> null
Step #11 - "cleanup gke cluster":   - project_id       = "gke-ai-eco-dev" -> null
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destroying... [id=gke-ai-eco-dev/roles/container.defaultNodeServiceAccount/serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0]: Destroying... [id=gke-ai-eco-dev/roles/stackdriver.resourceMetadata.writer/serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0]: Destroying... [id=gke-ai-eco-dev/roles/monitoring.metricWriter/serviceAccount:tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Destroying... [id=projects/gke-ai-eco-dev/locations/us-east4/clusters/ml-8f4839d-794-3a9541ca-cluster/nodePools/gpu-pool]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Destroying... [id=projects/gke-ai-eco-dev/locations/us-east4/clusters/ml-8f4839d-794-3a9541ca-cluster/nodePools/cpu-pool]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-resourceMetadata-writer[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-nodeService_account[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_project_iam_member.cluster_service_account-metric_writer[0]: Destruction complete after 8s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 1m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 1m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 2m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 2m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 2m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 2m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 2m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 2m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 2m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 2m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 2m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 2m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 2m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 2m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 3m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 3m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 3m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/gpu-pool, 3m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["gpu-pool"]: Destruction complete after 3m11s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 3m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 3m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 3m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 3m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 4m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 4m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 4m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...94-3a9541ca-cluster/nodePools/cpu-pool, 4m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_node_pool.pools["cpu-pool"]: Destruction complete after 4m32s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Destroying... [id=projects/gke-ai-eco-dev/locations/us-east4/clusters/ml-8f4839d-794-3a9541ca-cluster]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 1m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 1m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 1m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 1m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 1m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 1m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 2m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 2m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 2m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 2m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 2m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 2m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 3m0s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 3m10s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 3m20s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 3m30s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 3m40s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Still destroying... [id=projects/gke-ai-eco-dev/locations/us-ea...usters/ml-8f4839d-794-3a9541ca-cluster, 3m50s elapsed]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_container_cluster.primary: Destruction complete after 3m52s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destroying... [id=projects/gke-ai-eco-dev/serviceAccounts/tf-gke-ml-8f4839d-794--ob8a@gke-ai-eco-dev.iam.gserviceaccount.com]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.google_service_account.cluster_service_account[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.random_string.cluster_service_account_suffix[0]: Destroying... [id=ob8a]
Step #11 - "cleanup gke cluster": module.public-gke-standard-cluster[0].module.gke.random_string.cluster_service_account_suffix[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_service_networking_connection.private_service_access[0]: Destroying... [id=https%3A%2F%2Fwww.googleapis.com%2Fcompute%2Fv1%2Fprojects%2Fgke-ai-eco-dev%2Fglobal%2Fnetworks%2Fml-8f4839d-794-3a9541ca-false:servicenetworking.googleapis.com]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-8f4839d-794-3a9541ca-false"]: Destroying... [id=projects/gke-ai-eco-dev/regions/us-east4/subnetworks/ml-8f4839d-794-3a9541ca-false]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_service_networking_connection.private_service_access[0]: Destruction complete after 0s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Destroying... [id=projects/gke-ai-eco-dev/global/addresses/google-managed-services-ml-8f4839d-794-3a9541ca-false]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-8f4839d-794-3a9541ca-false"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...networks/ml-8f4839d-794-3a9541ca-false, 10s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Still destroying... [id=projects/gke-ai-eco-dev/global/addresse...services-ml-8f4839d-794-3a9541ca-false, 10s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_global_address.google-managed-services-range[0]: Destruction complete after 11s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-8f4839d-794-3a9541ca-false"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...networks/ml-8f4839d-794-3a9541ca-false, 20s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-8f4839d-794-3a9541ca-false"]: Still destroying... [id=projects/gke-ai-eco-dev/regions/us-east...networks/ml-8f4839d-794-3a9541ca-false, 30s elapsed]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_subnetwork.subnetwork["us-east4/ml-8f4839d-794-3a9541ca-false"]: Destruction complete after 31s
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_network.network: Destroying... [id=projects/gke-ai-eco-dev/global/networks/ml-8f4839d-794-3a9541ca-false]
Step #11 - "cleanup gke cluster": module.custom-network[0].google_compute_network.network: Still destroying... [id=projects/gke-ai-eco-dev/global/networks/ml-8f4839d-794-3a9541ca-false, 10s elapsed]
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": Error: Error waiting for Deleting Network: The network resource 'projects/gke-ai-eco-dev/global/networks/ml-8f4839d-794-3a9541ca-false' is already being used by 'projects/gke-ai-eco-dev/global/firewalls/ml-8f4839d-794-3a9541ca-false-wisboqeuj6al4v4p26vj2d3n'
Step #11 - "cleanup gke cluster": 
Step #11 - "cleanup gke cluster": 
Finished Step #11 - "cleanup gke cluster"
ERROR: step exited with non-zero status: 1
Starting Step #12 - "check result"
Step #12 - "check result": Already have image (with digest): gcr.io/gke-ai-eco-dev/terraform
Finished Step #12 - "check result"
PUSH
DONE

Build Log: https://console.cloud.google.com/cloud-build/builds;region=us-central1/3a9541ca-2847-45c2-ad0b-b9f5acf4d5bb?project=142966328212