🚌 Convert TM data into bus vehicle events #425
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mzappitello
commented
Aug 20, 2024
Comment on lines
12
to
33
| def create_dt_from_sam( | ||
| service_date_col: pl.Expr, sam_time_col: pl.Expr | ||
| ) -> pl.Expr: | ||
| """ | ||
| add a seconds after midnight to a service date to create a datetime object. | ||
| seconds after midnight is in boston local time, convert it to utc. | ||
| """ | ||
| return ( | ||
| service_date_col.cast(pl.Datetime) + pl.duration(seconds=sam_time_col) | ||
| ).map_elements( | ||
| lambda x: BOSTON_TZ.localize(x).astimezone(UTC_TZ), | ||
| return_dtype=pl.Datetime, | ||
| ) |
There was a problem hiding this comment.
a nice utility function to convert seconds after midnight columns into datetime columns.
mzappitello
commented
Aug 20, 2024
| ) | ||
|
|
||
|
|
||
| def generate_tm_events(tm_files: List[str]) -> pl.DataFrame: |
There was a problem hiding this comment.
this will generate the tm vehicle events dataframe that can be joined with the gtfs rt dataframe to create contain all vehicle events we're aware of.
mzappitello
commented
Aug 20, 2024
|
|
||
|
|
||
| def get_daily_work_pieces(daily_work_piece_files: List[str]) -> pl.DataFrame: | ||
| """ |
There was a problem hiding this comment.
this will generate a dataframe of daily work pieces, who was driving what bus on a given day. there are a lot of caveats though since i can't seam to get the joins to work perfectly. they are noted in the comments.
We want to take Transit Master data from the springboard bucket and convert them into Bus Events that can be joined against GTFS Realtime data. This will give us additional information about when a but arrived at stops as well as when it hits non revenue timepoints. Create a new function that takes a list of Transit Master stop crossing parquet paths and joins it against Transit Master Geo Nodes, Routes, Trips, and Vehicle Tables. Adjust column names, cast them appropriately, and do some modification to make them useable by later stages in the pipeline. Add test files and test cases to ensure ingestion and transformation is happening as expected.
Process Daily Work Piece logs into a dataframe of operator / vehicle records tied to block ids, run ids, and trip ids that can be joined against bus vehicle events.
the remote files runtime utility was updated and merged on main while the tm to bus events branch was completed. after rebasing, this patch wires everything up correctly.
mzappitello
force-pushed
the
tm_to_bus_events_3
branch
from
40c2007 to
909796f
Compare
Coverage of commit
|
rymarczy
force-pushed
the
tm_to_bus_events_3
branch
from
ead40e0 to
12371c6
Compare
Coverage of commit
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🚌 Convert TM data into bus vehicle events
Convert Transit Master Stop Crossing parquet files and Daily Piece of Work parquet files into datafarmes describing Vehicle Events and Operator / Vehicle assignments.