Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

186 filter to date method in gtfsinstance #190

Merged
merged 14 commits into from
Nov 6, 2023

Conversation

r-leyshon
Copy link
Contributor

@r-leyshon r-leyshon commented Oct 23, 2023

Description

bbox_filter_gtfs optionally filters to a list of dates present within the gtfs calendar.

Fixes #186

Motivation and Context

Reducing GTFS archives to valid features (stops, shapes etc) on a given date appears to reduce the likelihood of encountering r5py errors when building transport network. It is currently thought that these errors are encountered due to specific fast travel issues, although further investigation and reproduction of the encountered error is ongoing.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

bbox_filter_gtfs will continue to work as before, although specifying a populated list will run the gtfs_kit.feed.restrict_to_date() logic.

How Has This Been Tested?

Defensive checks and test suite included. Also a test marked with runinteg to ensure the r5py trasport network can be built from the filtered GTFS file.

Test configuration details:

  • OS: macos
  • Python version: Python 3.9.13
  • Java version: openjdk 11.0.19 2023-04-18 LTS
  • Python management system: pip

Advice for reviewer

Upon submission I noticed the test runners failing. After reviewing the fails, it became apparent that the current pytest invocation was incorrect and breaking. The changes to the CI workflows fix that and remove duplicated tests in all-os.tests.yml, where tests were being run twice:

  1. pytest --runinteg
  2. pytest

Checklist:

  • My code follows the intended structure of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

Additional comments

@r-leyshon r-leyshon added the enhancement New feature or request label Oct 23, 2023
@r-leyshon r-leyshon added this to the sprint 5 end milestone Oct 23, 2023
@r-leyshon r-leyshon linked an issue Oct 23, 2023 that may be closed by this pull request
@r-leyshon r-leyshon marked this pull request as draft October 23, 2023 10:36
@codecov-commenter
Copy link

codecov-commenter commented Oct 23, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (8bddb14) 97.77% compared to head (c080745) 97.81%.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev     #190      +/-   ##
==========================================
+ Coverage   97.77%   97.81%   +0.03%     
==========================================
  Files          14       14              
  Lines        1171     1188      +17     
==========================================
+ Hits         1145     1162      +17     
  Misses         26       26              
Flag Coverage Δ
unittests 97.81% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
src/transport_performance/gtfs/gtfs_utils.py 100.00% <100.00%> (ø)
src/transport_performance/utils/defence.py 100.00% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@r-leyshon r-leyshon marked this pull request as ready for review October 23, 2023 13:32
@r-leyshon r-leyshon added the GTFS label Oct 23, 2023
@ethan-moss ethan-moss removed the request for review from SergioRec October 24, 2023 08:24
@CBROWN-ONS CBROWN-ONS self-assigned this Oct 24, 2023
Copy link
Collaborator

@CBROWN-ONS CBROWN-ONS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @r-leyshon . Great work on this. Everything is working as expected.
I have a few small comments however they aren't huge things to clear up.

On the r5py stuff, from eyeballing it, it looks good to me. We might however want to get someone else to quickly run the test on their machine to ensure it works.

src/transport_performance/gtfs/gtfs_utils.py Outdated Show resolved Hide resolved
src/transport_performance/gtfs/gtfs_utils.py Outdated Show resolved Hide resolved
src/transport_performance/gtfs/gtfs_utils.py Outdated Show resolved Hide resolved
@r-leyshon

This comment was marked as resolved.

@r-leyshon
Copy link
Contributor Author

@CBROWN-ONS I have implemented all your suggestions except the final comment, which unless I have misunderstood is already the case. Many thanks for the good catches once again. Did you feel you needed a more in depth review of the tests or does a passing CI suite do enough to demonstrate the behaviour is as expected?

@CBROWN-ONS
Copy link
Collaborator

@CBROWN-ONS I have implemented all your suggestions except the final comment, which unless I have misunderstood is already the case. Many thanks for the good catches once again. Did you feel you needed a more in depth review of the tests or does a passing CI suite do enough to demonstrate the behaviour is as expected?

Hi Rich. After reviewing your comments on the matter, I agree that we are fine not to accept a parameter for date format while filtering the GTFS.

Great work on the other changes. Regarding the r5py stuff, I'm happy to merge as the CI demonstrates that these tests pass and are operational. Let me know if you agree with this, and I will merge this into dev.

@r-leyshon
Copy link
Contributor Author

@CBROWN-ONS I have implemented all your suggestions except the final comment, which unless I have misunderstood is already the case. Many thanks for the good catches once again. Did you feel you needed a more in depth review of the tests or does a passing CI suite do enough to demonstrate the behaviour is as expected?

Hi Rich. After reviewing your comments on the matter, I agree that we are fine not to accept a parameter for date format while filtering the GTFS.

Great work on the other changes. Regarding the r5py stuff, I'm happy to merge as the CI demonstrates that these tests pass and are operational. Let me know if you agree with this, and I will merge this into dev.

Good to go from my end, thanks again.

@CBROWN-ONS CBROWN-ONS merged commit e6a700b into dev Nov 6, 2023
10 of 11 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 6, 2023
* feat: Optionally filter gtfs to date list

* refactor: Toml has no nonetype, so empty list used to specify no date filter

* test: Invalid datestring causes raise

* test: Raises if filter_dates values do not exist within gtfs calendar

* refactor: All in_pth values use gtfs fixture path constant

* test: Assert filtered net can build r5py network

* fix: Runinteg invocations in workflows needed fixing

* fix: Runner doesn't install osmosis so deselect all tests with osmosis deps

* feat: Changes on bristol run

* refactor: Defense checks & type hinting

* docs: Add docstring to internal

* chore: Move _validate_datestring to defence.py

* docs: Update docstring

---------

Co-authored-by: Richard Leyshon <[email protected]> e6a700b
ethan-moss added a commit that referenced this pull request Nov 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request GTFS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

filter_to_date() method in GtfsInstance
3 participants