-
Notifications
You must be signed in to change notification settings - Fork 15
Replace "development" by "cal-itp-data-infra-staging" for AIRFLOW_ENV to match correct values
#4439
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…uration for AIRFLOW_ENV [#4426]
|
Terraform plan in iac/cal-itp-data-infra/airflow/us Plan: 0 to add, 4 to change, 0 to destroy.Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~ update in-place
Terraform will perform the following actions:
# google_storage_bucket_object.calitp-composer["dags/dags.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer" {
!~ crc32c = "tQl0DA==" -> (known after apply)
!~ detect_md5hash = "B95K3uhC41c6uMX4cvS/GA==" -> "different hash"
!~ generation = 1757464207493746 -> (known after apply)
id = "calitp-composer-dags/dags.py"
!~ md5hash = "B95K3uhC41c6uMX4cvS/GA==" -> (known after apply)
name = "dags/dags.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer["dags/download_gtfs_schedule_v2/_email_download_failures.py"] will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer" {
!~ crc32c = "W7LJjA==" -> (known after apply)
!~ detect_md5hash = "iRJ0PWcukGt+i1vBoZ/mmg==" -> "different hash"
!~ generation = 1760041506297314 -> (known after apply)
id = "calitp-composer-dags/download_gtfs_schedule_v2/_email_download_failures.py"
!~ md5hash = "iRJ0PWcukGt+i1vBoZ/mmg==" -> (known after apply)
name = "dags/download_gtfs_schedule_v2/_email_download_failures.py"
# (17 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-catalog will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-catalog" {
!~ content = (sensitive value)
!~ crc32c = "HFLpFA==" -> (known after apply)
!~ detect_md5hash = "AxlvqGlF0XQytL7X6uBrow==" -> "different hash"
!~ generation = 1761674676592506 -> (known after apply)
id = "calitp-composer-data/warehouse/target/catalog.json"
!~ md5hash = "AxlvqGlF0XQytL7X6uBrow==" -> (known after apply)
name = "data/warehouse/target/catalog.json"
# (16 unchanged attributes hidden)
}
# google_storage_bucket_object.calitp-composer-manifest will be updated in-place
!~ resource "google_storage_bucket_object" "calitp-composer-manifest" {
!~ content = (sensitive value)
!~ crc32c = "JvfK6A==" -> (known after apply)
!~ detect_md5hash = "/ncG1pOau+Ob3onRgEJBiQ==" -> "different hash"
!~ generation = 1761674678163486 -> (known after apply)
id = "calitp-composer-data/warehouse/target/manifest.json"
!~ md5hash = "/ncG1pOau+Ob3onRgEJBiQ==" -> (known after apply)
name = "data/warehouse/target/manifest.json"
# (16 unchanged attributes hidden)
}
Plan: 0 to add, 4 to change, 0 to destroy.📝 Plan generated in Plan Terraform for Warehouse and DAG changes #884 |
|
The following changes will be applied to the production Kubernetes cluster upon merge. BE AWARE this may not reveal changes that have been manually applied to the cluster getting undone—applying manual changes to the cluster should be avoided. diff -u -N /tmp/LIVE-4254321978/v1.ConfigMap.gtfs-rt-v3-test.archiver-channel-vars /tmp/MERGED-3958363687/v1.ConfigMap.gtfs-rt-v3-test.archiver-channel-vars
--- /tmp/LIVE-4254321978/v1.ConfigMap.gtfs-rt-v3-test.archiver-channel-vars 2025-10-29 05:01:23.763205669 +0000
+++ /tmp/MERGED-3958363687/v1.ConfigMap.gtfs-rt-v3-test.archiver-channel-vars 2025-10-29 05:01:23.764205668 +0000
@@ -1,6 +1,6 @@
apiVersion: v1
data:
- AIRFLOW_ENV: development
+ AIRFLOW_ENV: cal-itp-data-infra-staging
CALITP_BUCKET__GTFS_DOWNLOAD_CONFIG: gs://test-calitp-gtfs-download-config
CALITP_BUCKET__GTFS_RT_RAW: gs://test-calitp-gtfs-rt-raw-v2
CALITP_HUEY_CONSUMER_WORKERS: "8" |
lauriemerrell
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One comment, otherwise LGTM. I had made the fix in dags.py but I can just rebase on this
lauriemerrell
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ope, missed a save -- here's the comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As noted in the ticket, I am just not sure what this env var does in this context, not sure if there's been a functional issue on the test archiver?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, I haven't work with that yet. I asked about your tests to see if any were related to this on.
We definitely need to test it before merging this change.
Description
This PR fixes the different cases where
AIRFLOW_ENVis set to "cal-itp-data-infra-staging" but some tests are checking for "development" instead.Resolves [#4426]
Type of change
How has this been tested?
Post-merge follow-ups