Scrape geoportal for shn, add SHN flag on stops table #3529

charlie-costanzo · 2024-11-05T17:53:17Z

Description

As requested in #3397, this PR creates:

An airflow operator to scrape the state geoportal for the state highway network geometry, clean the data and save it to a GCS bucket
A dag task for that operator
A dag task for creating a new external table for the SHN geometry
New source, staging, and docs in dbt for this data
- Geometry operations in the dim_stops_latest table and a new flag column on the table to indicate whether it exists on the state highway network

Resolves #3397

Type of change

New feature

How has this been tested?

locally using airflow/dbt

Post-merge follow-ups

Actions required (specified below)
GCS bucket needs to be added to Composer variables, new airflow operator needs to be turned on

github-actions · 2024-11-12T16:50:03Z

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

For new models, do they all have a surrogate primary key that is tested to be not-null and unique?

New models 🌱

calitp_warehouse.staging.state_geoportal.stg_state_geoportal__state_highway_network_stops

DAG

Legend (in order of precedence)

Resource type	Indicator	Resolution
Large table-materialized model	Orange	Make the model incremental
Large model without partitioning or clustering	Orange	Add partitioning and/or clustering
View with more than one child	Yellow	Materialize as a table or incremental
Incremental	Light green
Table	Green
View	White

mjumbewu

Ok, I know it looks like a lot of comments, but most of it pertains to the options on the geoportal operator. Overall, I really dig this work. Also note that I did not run any of the suggested changes, so 🙏 please make sure they make sense. Happy to hop on a call tomorrow to discuss.

airflow/dags/scrape_state_geoportal/state_highway_network.yml

mjumbewu · 2024-11-14T22:58:42Z

airflow/plugins/operators/scrape_state_geoportal.py

+    product: str
+    where: str
+    outFields: str
+    f: str


suggestion: We probably always want the format that comes from the API to be the same (e.g. GeoJSON) if this operator is going to convert it to JSON-L, so maybe we don't include an f option.

airflow/plugins/operators/scrape_state_geoportal.py

mjumbewu · 2024-11-14T23:39:10Z

airflow/plugins/operators/scrape_state_geoportal.py

+        if self.product == "state_highway_network":
+            # Select and rename columns
+            columns = {
+                "properties.Route": "Route",
+                "properties.County": "County",
+                "properties.District": "District",
+                "properties.RouteType": "RouteType",
+                "properties.Direction": "Direction",
+                "geometry.type": "type",
+                "geometry.coordinates": "coordinates",
+            }
+            df = df[list(columns.keys())].rename(columns=columns)


suggestion (non-blocking): I don't think this is something that needs to be addressed right now, but it seems like we could do this without hard-coding the product here. We're really just cleaning up column names -- removing properties. and geometry. prefixes that come from the GeoJSON.

yes very good point, refactored

mjumbewu · 2024-11-14T23:40:19Z

warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql

+
+stg_state_geoportal__state_highway_network_stops AS (
+SELECT *
+-- FROM `cal-itp-data-infra-staging.external_state_geoportal.stg_state_geoportal__state_highway_network_stops`


Suggested change

-- FROM `cal-itp-data-infra-staging.external_state_geoportal.stg_state_geoportal__state_highway_network_stops`

warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql

evansiroky · 2024-11-19T19:16:51Z

warehouse/models/mart/gtfs_schedule_latest/dim_stops_latest.sql

+    IF(stops_on_shn.stop_id IS NOT NULL, TRUE, FALSE) AS exists_in_dim_stops_latest
+FROM
+    dim_stops_latest
+LEFT JOIN
+    stops_on_shn
+ON
+    dim_stops_latest.stop_id = stops_on_shn.stop_id


Is joining on stop_id going to potentially cause some conflicts here? There may be two feeds that have the same stop_id that this could match on. Would a join on dim_stops_latest._gtfs_key be more appropriate?

Hey @evansiroky – you're completely right. I must have left this in here after some comparative testing, apologies! I just made the swap and am re-requesting your review

mjumbewu

Ok, just two more small notes from me, but otherwise I'm ready to approve! Lemme know if you have any questions.

mjumbewu · 2024-11-21T22:14:27Z

airflow/dags/scrape_state_geoportal/state_highway_network.yml

+where: "1=1"  # You can change this to filter data
+outFields: "*"  # Specify the fields to return
+f: "geojson"  # Format of the response


suggestion: These are no longer necessary with the new defaults, right? Should probably remove them (but verify that it still works with them removed).

removed and still ran successfully!

mjumbewu · 2024-11-21T22:37:44Z

airflow/plugins/operators/scrape_state_geoportal.py

+                params["resultOffset"] = offset
+
+                # Make the request
+                response = requests.get(validated_url, params=params)


question: What should happen if we get a non-200 response from the server? I would think we should raise an exception so that the task fails and has to retry. Does that seem right? You could add .raise_for_status() at the end of the line if so. See https://3.python-requests.org/user/quickstart/#response-status-codes

added! great suggestion

evansiroky

Approved pending your discussion with @mjumbewu.

…oordinates

…hen ready

Co-authored-by: Mjumbe Poe <[email protected]>

…uffer distance

charlie-costanzo · 2024-11-26T17:38:08Z

CALITP_BUCKET__STATE_GEOPORTAL_DATA_PRODUCTS variable added to composer environment

charlie-costanzo self-assigned this Nov 5, 2024

charlie-costanzo requested review from evansiroky, vevetron and tiffanychu90 as code owners November 5, 2024 17:53

charlie-costanzo marked this pull request as draft November 5, 2024 18:00

charlie-costanzo force-pushed the shn-stops-attribute branch from 61a59c9 to 4a914c0 Compare November 12, 2024 17:37

mjumbewu self-requested a review November 12, 2024 20:41

charlie-costanzo force-pushed the shn-stops-attribute branch from 6278fde to c7384bc Compare November 13, 2024 16:35

charlie-costanzo mentioned this pull request Nov 13, 2024

airflow: PythonOperator to scrape state highway network stop locations #3485

Closed

6 tasks

mjumbewu reviewed Nov 14, 2024

View reviewed changes

charlie-costanzo force-pushed the shn-stops-attribute branch 2 times, most recently from fd5e5d4 to 4365a6c Compare November 18, 2024 20:45

charlie-costanzo marked this pull request as ready for review November 18, 2024 20:45

charlie-costanzo added the data-pipeline-ingestion-and-modeling Ingesting, parsing and modeling data. Evan Siroky is product owner. label Nov 18, 2024

evansiroky reviewed Nov 19, 2024

View reviewed changes

charlie-costanzo force-pushed the shn-stops-attribute branch from 4365a6c to 4e6ecbb Compare November 20, 2024 18:34

charlie-costanzo requested review from evansiroky and mjumbewu November 20, 2024 18:45

mjumbewu approved these changes Nov 21, 2024

View reviewed changes

evansiroky approved these changes Nov 22, 2024

View reviewed changes

charlie-costanzo force-pushed the shn-stops-attribute branch 4 times, most recently from f0247fe to ffc8e3a Compare November 26, 2024 14:40

charlie-costanzo added 5 commits November 26, 2024 12:29

scrape geoportal for shn, add flag on stops table

a807a10

successful initial scrape of geoportal endpoint

e8f3ad9

functioning json format and external table creation, need to modify c…

44fa45f

…oordinates

shn external tables

ac47b74

add initial source, staging, and docs for state geoportal scrape

0af1759

charlie-costanzo and others added 15 commits November 26, 2024 12:29

add flag for shn to dim_stops_latest

df234a4

readd templating for buckets and remove hardcode schema

68d3fac

refactor operator to be more agnostic for data inputs

6a9064c

add handling for state_highway_network table

386abf8

fix comment

f9b3615

remove local variables and comments to allow for a production merge w…

0dd22f7

…hen ready

use stop_id for calculations

32becb7

Apply suggestions from code review

45019d1

Co-authored-by: Mjumbe Poe <[email protected]>

restore source table

008bc0e

remove hardcoded source, remove old comments, add comment decribing b…

553682c

…uffer distance

remove hardcoded bucket

cf5f01e

refactor how we keep final columns, dynamically rename columns

11d2c12

restore _gtfs_key over stop_id in shn geo calculation

791ffef

checking before refactor

2bcea54

revisions based on Mjumbe's review

0ca4a89

charlie-costanzo force-pushed the shn-stops-attribute branch from ffc8e3a to 0ca4a89 Compare November 26, 2024 17:30

charlie-costanzo merged commit 0dfe3b6 into main Nov 26, 2024
4 checks passed

charlie-costanzo deleted the shn-stops-attribute branch November 26, 2024 17:38

charlie-costanzo restored the shn-stops-attribute branch November 26, 2024 19:01

charlie-costanzo mentioned this pull request Nov 26, 2024

revise state geoportal scrape operator import #3563

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scrape geoportal for shn, add SHN flag on stops table #3529

Scrape geoportal for shn, add SHN flag on stops table #3529

charlie-costanzo commented Nov 5, 2024 •

edited

Loading

github-actions bot commented Nov 12, 2024 •

edited

Loading

mjumbewu left a comment

mjumbewu Nov 14, 2024

mjumbewu Nov 14, 2024

charlie-costanzo Nov 18, 2024

mjumbewu Nov 14, 2024

evansiroky Nov 19, 2024

charlie-costanzo Nov 20, 2024

mjumbewu left a comment

mjumbewu Nov 21, 2024

charlie-costanzo Nov 22, 2024

mjumbewu Nov 21, 2024

charlie-costanzo Nov 22, 2024

evansiroky left a comment •

edited

Loading

charlie-costanzo commented Nov 26, 2024

Scrape geoportal for shn, add SHN flag on stops table #3529

Scrape geoportal for shn, add SHN flag on stops table #3529

Conversation

charlie-costanzo commented Nov 5, 2024 • edited Loading

Description

Type of change

How has this been tested?

Post-merge follow-ups

github-actions bot commented Nov 12, 2024 • edited Loading

Checks/potential follow-ups

New models 🌱

DAG

mjumbewu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjumbewu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evansiroky left a comment • edited Loading

Choose a reason for hiding this comment

charlie-costanzo commented Nov 26, 2024

charlie-costanzo commented Nov 5, 2024 •

edited

Loading

github-actions bot commented Nov 12, 2024 •

edited

Loading

evansiroky left a comment •

edited

Loading