Skip to content

Commit

Permalink
deploy cloud scheduler by default to reduce steps to start using DQM
Browse files Browse the repository at this point in the history
Change-Id: Ie9699804fff204305f4239a772f2feec62943072
  • Loading branch information
psnelG committed Jun 9, 2023
1 parent 5250086 commit 982bb07
Show file tree
Hide file tree
Showing 5 changed files with 25 additions and 3 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ You can read more in the [configuration docs](docs/config.md).

### [Usage](docs/usage.md)

DQM can be automated with [Cloud Scheduler](https://cloud.google.com/scheduler).
DQM can be automated with [Cloud Scheduler](https://cloud.google.com/scheduler). Simply set `pause_scheduler` to `false` in the .tfvars file during deployment.

It outputs extensive logging, which can be leveraged for notifications or dashboards.

Expand Down
6 changes: 6 additions & 0 deletions deployment/terraform/api.tf
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,10 @@ resource "google_project_service" "workflows" {
project = var.project_id
service = "workflows.googleapis.com"
disable_on_destroy = false
}

resource "google_project_service" "scheduler" {
project = var.project_id
service = "cloudscheduler.googleapis.com"
disable_on_destroy = false
}
7 changes: 5 additions & 2 deletions deployment/terraform/example.tfvars
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
project_id = "YOUR-GCP-PROJECT"
project_id = "YOUR-PROJECT-ID"
cloud_storage_region = "EU" # or US
bigquery_location = "EU" # or US
bigquery_location = "EU" # or US
workflow_region = "europe-west1"
cloud_function_region = "europe-west1"
service_account_name = "dqm-account"

pause_scheduler = true # scheduler is paused by default, set to "false" to enable
trigger_schedule_cron = "0 8 * * *" # daily execution
10 changes: 10 additions & 0 deletions deployment/terraform/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,13 @@ variable "bigquery_location" {
type = string
description = "region where to call BigQuery"
}

variable "trigger_schedule_cron" {
type = string
description = "cron defining how often cloud scheduler should trigger dqm"
}

variable "pause_scheduler" {
type = string
description = "true or false, depending if scheduler should automatically trigger the workflow"
}
3 changes: 3 additions & 0 deletions docs/install.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,12 @@ DQM is deployed using Terraform, which comes pre-installed on Google Cloud Shell
* `project_id`: GCP Project to deploy DQM onto
1. Optional:
* `cloud_storage_region`: Cloud Storage Bucket region to store configuration files
* `biguery`: Location where bigquery queries are executed
* `workflow_region`: Cloud Workflows region for DQM's workflow
* `cloud_function_region`: Cloud Function region for DQM's core
* `service_account_name`: Name for Service Account used by DQM
* `pause_scheduler`: Set to false to enable cloud scheduler
* `trigger_schedule_cron`: Cron schedule to run DQM if scheduler is not paused
1. Run `terraform init`
1. Run `terraform plan -var-file="example.tfvars"`
1. If prompted, review the `gcloud` authentication prompt and click "Authorize".
Expand Down

0 comments on commit 982bb07

Please sign in to comment.