Skip to content

Conversation

@erikamov
Copy link
Contributor

Description

This PR updates the data type definition of external_blackcat.all_ntdreports columns (NTD Blackcat reports 2025) requested on cal-itp/ntd-modernization#8.

The Notes column was missing a s at the end, so it will be fixed with this PR.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Tested the creation of the external table through local Airflow:

image

And downstream tables

❯ poetry run dbt run -s source:calitp_warehouse.ntd_report_validation.all_ntdreports+ --target staging
21:02:06  Running with dbt=1.10.1
21:02:08  Registered adapter: bigquery=1.10.0
21:02:09  Found 621 models, 1241 data tests, 14 seeds, 225 sources, 4 exposures, 1053 macros
21:02:09
21:02:09  Concurrency: 8 threads (target='staging')
21:02:09
21:02:13  1 of 17 START sql view model staging.stg_ntd_a10 ............................... [RUN]
21:02:13  2 of 17 START sql view model staging.stg_ntd_a30_assetandresourceinfo .......... [RUN]
21:02:13  3 of 17 START sql view model staging.stg_ntd_rr20_rural ........................ [RUN]
21:02:13  4 of 17 START sql view model staging.stg_ntd_rr20_urban_tribal ................. [RUN]
21:02:14  1 of 17 OK created sql view model staging.stg_ntd_a10 .......................... [CREATE VIEW (0 processed) in 0.96s]
21:02:14  4 of 17 OK created sql view model staging.stg_ntd_rr20_urban_tribal ............ [CREATE VIEW (0 processed) in 0.96s]
21:02:14  3 of 17 OK created sql view model staging.stg_ntd_rr20_rural ................... [CREATE VIEW (0 processed) in 0.96s]
21:02:14  5 of 17 START sql view model staging.int_ntd_a10_facilitiesdata ................ [RUN]
21:02:14  6 of 17 START sql view model staging.int_ntd_rr20_financial_fare_revenues ...... [RUN]
21:02:14  7 of 17 START sql view model staging.int_ntd_rr20_financial_specific_funds ..... [RUN]
21:02:14  8 of 17 START sql view model staging.int_ntd_rr20_financial_total_exp .......... [RUN]
21:02:14  9 of 17 START sql table model staging.int_ntd_rr20_service_1alldata ............ [RUN]
21:02:14  2 of 17 OK created sql view model staging.stg_ntd_a30_assetandresourceinfo ..... [CREATE VIEW (0 processed) in 0.99s]
21:02:15  8 of 17 OK created sql view model staging.int_ntd_rr20_financial_total_exp ..... [CREATE VIEW (0 processed) in 0.97s]
21:02:15  6 of 17 OK created sql view model staging.int_ntd_rr20_financial_fare_revenues . [CREATE VIEW (0 processed) in 0.97s]
21:02:15  10 of 17 START sql view model mart_ntd_validation.fct_ntd_rr20_equal_totals_check  [RUN]
21:02:15  7 of 17 OK created sql view model staging.int_ntd_rr20_financial_specific_funds  [CREATE VIEW (0 processed) in 1.06s]
21:02:15  5 of 17 OK created sql view model staging.int_ntd_a10_facilitiesdata ........... [CREATE VIEW (0 processed) in 1.07s]
21:02:15  11 of 17 START sql view model mart_ntd_validation.fct_ntd_rr20_funds_checks .... [RUN]
21:02:15  12 of 17 START sql view model mart_ntd_validation.fct_ntd_a10_facilitiescheck .. [RUN]
21:02:16  10 of 17 OK created sql view model mart_ntd_validation.fct_ntd_rr20_equal_totals_check  [CREATE VIEW (0 processed) in 1.00s]
21:02:16  11 of 17 OK created sql view model mart_ntd_validation.fct_ntd_rr20_funds_checks  [CREATE VIEW (0 processed) in 1.00s]
21:02:16  12 of 17 OK created sql view model mart_ntd_validation.fct_ntd_a10_facilitiescheck  [CREATE VIEW (0 processed) in 1.00s]
21:02:17  9 of 17 OK created sql table model staging.int_ntd_rr20_service_1alldata ....... [CREATE TABLE (0.0 rows, 91.9 MiB processed) in 2.98s]
21:02:17  13 of 17 START sql view model staging.int_ntd_a30_voms_vins_totals ............. [RUN]
21:02:17  14 of 17 START sql view model staging.int_ntd_rr20_service_2ratioslong ......... [RUN]
21:02:18  14 of 17 OK created sql view model staging.int_ntd_rr20_service_2ratioslong .... [CREATE VIEW (0 processed) in 0.73s]
21:02:18  15 of 17 START sql table model staging.int_ntd_rr20_service_3ratios_wide ....... [RUN]
21:02:18  13 of 17 OK created sql view model staging.int_ntd_a30_voms_vins_totals ........ [CREATE VIEW (0 processed) in 0.97s]
21:02:18  16 of 17 START sql view model mart_ntd_validation.fct_ntd_a30_vomscheck ........ [RUN]
21:02:19  16 of 17 OK created sql view model mart_ntd_validation.fct_ntd_a30_vomscheck ... [CREATE VIEW (0 processed) in 1.00s]
21:02:21  15 of 17 OK created sql table model staging.int_ntd_rr20_service_3ratios_wide .. [CREATE TABLE (0.0 rows, 0 processed) in 2.93s]
21:02:21  17 of 17 START sql view model mart_ntd_validation.fct_ntd_rr20_service_checks .. [RUN]
21:02:22  17 of 17 OK created sql view model mart_ntd_validation.fct_ntd_rr20_service_checks  [CREATE VIEW (0 processed) in 0.87s]
21:02:22
21:02:22  Finished running 2 table models, 15 view models in 0 hours 0 minutes and 12.66 seconds (12.66s).
21:02:22
21:02:22  Completed successfully
21:02:22
21:02:22  Done. PASS=17 WARN=0 ERROR=0 SKIP=0 NO-OP=0 TOTAL=17

Post-merge follow-ups

  • No action required
  • Actions required (specified below)

Fix `Notes` column name missing `S` at the end.
@github-actions
Copy link

Terraform plan in iac/cal-itp-data-infra-staging/airflow/us

Plan: 0 to add, 1 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-staging-composer["dags/create_external_tables/ntd_report_validation/external_table_all_ntdreports.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-staging-composer" {
!~      crc32c              = "JiH76w==" -> (known after apply)
!~      detect_md5hash      = "y+zH5zi1iBckCS9dcMXqwQ==" -> "different hash"
!~      generation          = 1752623512996143 -> (known after apply)
        id                  = "calitp-staging-composer-dags/create_external_tables/ntd_report_validation/external_table_all_ntdreports.yml"
!~      md5hash             = "y+zH5zi1iBckCS9dcMXqwQ==" -> (known after apply)
        name                = "dags/create_external_tables/ntd_report_validation/external_table_all_ntdreports.yml"
#        (17 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #889

@github-actions
Copy link

Terraform plan in iac/cal-itp-data-infra/airflow/us

Plan: 0 to add, 3 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
!~  update in-place

Terraform will perform the following actions:

  # google_storage_bucket_object.calitp-composer["dags/create_external_tables/ntd_report_validation/external_table_all_ntdreports.yml"] will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer" {
!~      crc32c              = "JiH76w==" -> (known after apply)
!~      detect_md5hash      = "y+zH5zi1iBckCS9dcMXqwQ==" -> "different hash"
!~      generation          = 1752703531845452 -> (known after apply)
        id                  = "calitp-composer-dags/create_external_tables/ntd_report_validation/external_table_all_ntdreports.yml"
!~      md5hash             = "y+zH5zi1iBckCS9dcMXqwQ==" -> (known after apply)
        name                = "dags/create_external_tables/ntd_report_validation/external_table_all_ntdreports.yml"
#        (17 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-catalog will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-catalog" {
!~      content             = (sensitive value)
!~      crc32c              = "Z+nINw==" -> (known after apply)
!~      detect_md5hash      = "oztTJY65DeqKvihHgQQiAQ==" -> "different hash"
!~      generation          = 1761769654941394 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/catalog.json"
!~      md5hash             = "oztTJY65DeqKvihHgQQiAQ==" -> (known after apply)
        name                = "data/warehouse/target/catalog.json"
#        (16 unchanged attributes hidden)
    }

  # google_storage_bucket_object.calitp-composer-manifest will be updated in-place
!~  resource "google_storage_bucket_object" "calitp-composer-manifest" {
!~      content             = (sensitive value)
!~      crc32c              = "qfwvow==" -> (known after apply)
!~      detect_md5hash      = "1cJlOr5NQvragutOKHNfgg==" -> "different hash"
!~      generation          = 1761769656597758 -> (known after apply)
        id                  = "calitp-composer-data/warehouse/target/manifest.json"
!~      md5hash             = "1cJlOr5NQvragutOKHNfgg==" -> (known after apply)
        name                = "data/warehouse/target/manifest.json"
#        (16 unchanged attributes hidden)
    }

Plan: 0 to add, 3 to change, 0 to destroy.

📝 Plan generated in Plan Terraform for Warehouse and DAG changes #889

Copy link
Member

@charlie-costanzo charlie-costanzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for taking care of this!

@erikamov erikamov merged commit b4ff8d9 into main Nov 3, 2025
8 checks passed
@erikamov erikamov deleted the mov/8-ntd-validation-2025 branch November 3, 2025 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants