Skip to content

Conversation

@erikamov
Copy link
Contributor

Description

This PR fixes the different cases where AIRFLOW_ENV is set to "cal-itp-data-infra-staging" but some tests are checking for "development" instead.

Resolves [#4426]

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Post-merge follow-ups

  • No action required
  • Actions required (specified below)

@github-actions
Copy link

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 4 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer["dags/dags.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "tQl0DA==" -> (known after apply)
!~      detect_md5hash      = "B95K3uhC41c6uMX4cvS/GA==" -> "different hash"
!~      generation          = 1757464207493746 -> (known after apply)
        id                  = "calitp-composer-dags/dags.py"
!~      md5hash             = "B95K3uhC41c6uMX4cvS/GA==" -> (known after apply)
        name                = "dags/dags.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer["dags/download_gtfs_schedule_v2/_email_download_failures.py"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "W7LJjA==" -> (known after apply)
!~      detect_md5hash      = "iRJ0PWcukGt+i1vBoZ/mmg==" -> "different hash"
!~      generation          = 1760041506297314 -> (known after apply)
        id                  = "calitp-composer-dags/download_gtfs_schedule_v2/_email_download_failures.py"
!~      md5hash             = "iRJ0PWcukGt+i1vBoZ/mmg==" -> (known after apply)
        name                = "dags/download_gtfs_schedule_v2/_email_download_failures.py"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "HFLpFA==" -> (known after apply)
!~      detect_md5hash      = "AxlvqGlF0XQytL7X6uBrow==" -> "different hash"
!~      generation          = 1761674676592506 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "AxlvqGlF0XQytL7X6uBrow==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "JvfK6A==" -> (known after apply)
!~      detect_md5hash      = "/ncG1pOau+Ob3onRgEJBiQ==" -> "different hash"
!~      generation          = 1761674678163486 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "/ncG1pOau+Ob3onRgEJBiQ==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 4 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #884

@github-actions
Copy link

github-actions bot commented Oct 29, 2025

The following changes will be applied to the production Kubernetes cluster upon merge.

BE AWARE this may not reveal changes that have been manually applied to the cluster getting undone—applying manual changes to the cluster should be avoided.

diff -u -N /tmp/LIVE-4254321978/v1.ConfigMap.gtfs-rt-v3-test.archiver-channel-vars /tmp/MERGED-3958363687/v1.ConfigMap.gtfs-rt-v3-test.archiver-channel-vars
--- /tmp/LIVE-4254321978/v1.ConfigMap.gtfs-rt-v3-test.archiver-channel-vars	2025-10-29 05:01:23.763205669 +0000
+++ /tmp/MERGED-3958363687/v1.ConfigMap.gtfs-rt-v3-test.archiver-channel-vars	2025-10-29 05:01:23.764205668 +0000
@@ -1,6 +1,6 @@
 apiVersion: v1
 data:
-  AIRFLOW_ENV: development
+  AIRFLOW_ENV: cal-itp-data-infra-staging
   CALITP_BUCKET__GTFS_DOWNLOAD_CONFIG: gs://test-calitp-gtfs-download-config
   CALITP_BUCKET__GTFS_RT_RAW: gs://test-calitp-gtfs-rt-raw-v2
   CALITP_HUEY_CONSUMER_WORKERS: "8"

Copy link
Contributor

@lauriemerrell lauriemerrell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One comment, otherwise LGTM. I had made the fix in dags.py but I can just rebase on this

Copy link
Contributor

@lauriemerrell lauriemerrell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ope, missed a save -- here's the comment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted in the ticket, I am just not sure what this env var does in this context, not sure if there's been a functional issue on the test archiver?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I haven't work with that yet. I asked about your tests to see if any were related to this on.
We definitely need to test it before merging this change.

@erikamov erikamov marked this pull request as draft October 30, 2025 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants