Fixing issue with Vertex AI Model Connection on BigQuery (#253)

* predicting for only the users with traffic in the past 72h - purchase propensity * running inference only for users events in the past 72h * including 72h users for all models predictions * considering null values in TabWorkflow models * deleting unused pipfile * upgrading lib versions * implementing reporting preprocessing as a new pipeline * adding more code documentation * adding important information on the main README.md and DEVELOPMENT.md * adding schedule run name and more code documentation * implementing a new scheduler using the vertex ai sdk & adding user_id to procedures for consistency * adding more code documentation * adding code doc to the python custom component * adding more code documentation * fixing aggregated predictions query * removing unnecessary resources from deployment * Writing MDS guide * adding the MDS developer and troubleshooting documentation * fixing deployment for activation pipelines and gemini dataset * Update README.md * Update README.md * Update README.md * Update README.md * removing deprecated api * fixing purchase propensity pipelines names * adding extra condition for when there is not enough data for the window interval to be applied on backfill procedures * adding more instructions for post deployment and fixing issues when GA4 export was configured for less than 10 days * removing unnecessary comments * adding the number of past days to process in the variables files * adding comment about combining data from different ga4 export datasets to data store * fixing small issues with feature engineering and ml pipelines * fixing hyper parameter tuning for kmeans modeling * fixing optuna parameters * adding cloud shell image * fixing the list of all possible users in the propensity training preparation tables * additional guardrails for when there is not enough data * adding more documentation * adding more doc to feature store * add feature store documentation * adding ml pipelines docs * adding ml pipelines docs * adding more documentation * adding user agent client info * fixing scope of client info * fix * removing client_info from vertex components * fixing versioning of tf submodules * reconfiguring meta providers * fixing issue 187 * chore(deps): upgrade terraform providers and modules version * chore(deps): set the provider version * chore: formatting * fix: brand naming * fix: typo * fixing secrets issue * implementing secrets region as tf variable * implementing secrets region as tf variable * last changes requested by lgrangeau * documenting keys location better * implementing vpc peering network * Update README.md * Rebase Main into Multi-property (#243) * Update README.md * ensure the build bucket is created in the specified region (#230) * Update audience_segmentation_query_template.sqlx * Update auto_audience_segmentation_query_template.sqlx * Update churn_propensity_query_template.sqlx * Update cltv_query_template.sqlx * Update purchase_propensity_query_template.sqlx * Restrict regions for GCP Cloud Build support (#241) * Update README.md * Move to uv (#242) * add uv required project table segment in toml file * switch to uv in terraform deployment * switch to uv * remove poetry usage from terraform * format * remove poetry * Add files via upload --------- Co-authored-by: Charlie Wang <[email protected]> Co-authored-by: Mårten Lindblad <[email protected]> * supporting property id in the resources * fixing iam member roles issues * fixing issue with service account iam resources * fixing issue with connection between vertex and bq --------- Co-authored-by: Carlos Timoteo <[email protected]> Co-authored-by: Laurent Grangeau <[email protected]> Co-authored-by: Charlie Wang <[email protected]> Co-authored-by: Mårten Lindblad <[email protected]>
GoogleCloudPlatform · Nov 22, 2024 · 1631315 · 1631315
1 parent b9b127f
commit 1631315
Show file tree

Hide file tree

Showing 2 changed files with 15 additions and 1 deletion.
diff --git a/infrastructure/terraform/modules/feature-store/bigquery-procedures.tf b/infrastructure/terraform/modules/feature-store/bigquery-procedures.tf
@@ -1487,6 +1487,7 @@ resource "null_resource" "check_gemini_model_exists" {
   triggers = {
     vertex_ai_connection_exists = google_bigquery_connection.vertex_ai_connection.id
     gemini_model_created        = null_resource.create_gemini_model.id
+    role_propagated             = time_sleep.wait_for_vertex_ai_connection_sa_role_propagation.id
   }
 
   provisioner "local-exec" {
@@ -1509,7 +1510,9 @@ resource "null_resource" "check_gemini_model_exists" {
 
   depends_on = [
     google_bigquery_connection.vertex_ai_connection,
-    null_resource.create_gemini_model
+    null_resource.create_gemini_model,
+    time_sleep.wait_for_vertex_ai_connection_sa_role_propagation,
+    google_project_iam_member.vertex_ai_connection_sa_roles
   ]
 }
 

diff --git a/infrastructure/terraform/modules/feature-store/main.tf b/infrastructure/terraform/modules/feature-store/main.tf
@@ -148,6 +148,17 @@ resource "google_project_iam_member" "vertex_ai_connection_sa_roles" {
   }
 }
 
+# Propagation time for change of access policy typically takes 2 minutes
+# according to https://cloud.google.com/iam/docs/access-change-propagation
+# this wait make sure the policy changes are propagated before proceeding
+# with the build
+resource "time_sleep" "wait_for_vertex_ai_connection_sa_role_propagation" {
+  create_duration = "120s"
+  depends_on = [
+    google_project_iam_member.vertex_ai_connection_sa_roles
+  ]
+}
+
 
 #module "vertex_ai_connection_sa_roles" {
 #  source  = "terraform-google-modules/iam/google//modules/member_iam"