Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

first draft for implementing link vehicle capacity counts handler #189

Merged
merged 10 commits into from
Apr 6, 2022

Conversation

elizabethc-arup
Copy link
Contributor

This is linked to the link vehicle capacity feature request described in the issue #187, it adapts the link volume counts handler and adds vehicle capacities instead of counts so that an event like <event time="25201.0" type="entered link" vehicle="bus1" link="2-3" /> will result in the bus vehicle capacity for bus1 added to link 2-3.

For pt vehicles, the vehicle capacity is determined by mapping the vehicle ids to capacities using the functions in the TransitVehicles class in inputs.py and for cars, a default capacity value of 5 is used.

elara/elara/inputs.py

Lines 406 to 468 in 9a68ece

class TransitVehicles(InputTool):
requirements = ['transit_vehicles_path']
veh_type_mode_map = None
veh_type_capacity_map = None
types = None
veh_id_veh_type_map = None
transit_vehicle_counts = None
def build(self, resources: dict, write_path: Optional[str] = None):
"""
Transit vehicles object constructor.
:param resources: dict, supplier resources
:param write_path: Optional output path overwrite
"""
super().build(resources)
path = resources['transit_vehicles_path'].path
# Vehicle types to mode correspondence
self.veh_type_mode_map = {
"Rail": "train",
"Suburban Railway": "suburban rail",
"Underground Service": "subway",
"Metro Service": "subway",
"Bus": "bus",
"Coach Service": "coach",
"Tram": "tram",
"Ferry": "ferry",
"Gondola": "gondola"
}
self.logger.debug(f'veh type mode map = {self.veh_type_mode_map}')
# Vehicle type to total capacity correspondence
self.veh_type_capacity_map = dict(
[
self.transform_veh_type_elem(elem)
for elem in get_elems(path, "vehicleType")
]
)
self.logger.debug(f'veh type capacity map = {self.veh_type_capacity_map}')
# Vehicle ID to vehicle type correspondence
self.logger.debug('Building veh id to veh type map')
self.veh_id_veh_type_map = {
elem.get("id"): elem.get("type") for elem in get_elems(path, "vehicle")
}
self.types, self.transit_vehicle_counts = count_values(self.veh_id_veh_type_map)
self.logger.debug(f'veh types = {self.types}')
@staticmethod
def transform_veh_type_elem(elem):
"""
Extract the vehicle type and total capacity from a vehicleType XML element.
:param elem: vehicleType XML element
:return: (vehicle type, capacity) tuple
"""
strip_namespace(elem) # TODO only needed for this input = janky
ident = elem.xpath("@id")[0]
seated_capacity = float(elem.xpath("capacity/seats/@persons")[0])
standing_capacity = float(elem.xpath("capacity/standingRoom/@persons")[0])
return ident, seated_capacity + standing_capacity

Currently, an if statement is used to deal with the transit_vehicle.xml version/legacy issue described in #188 as a temporary solution, it would be great if I could get some advice also on how best to resolve this. Thank you :)

@elizabethc-arup elizabethc-arup requested review from fredshone, andkay and mfitz and removed request for fredshone and andkay March 7, 2022 01:18
@andkay andkay self-requested a review March 7, 2022 10:16
fredshone
fredshone previously approved these changes Mar 7, 2022
Copy link
Collaborator

@fredshone fredshone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Interesting concept.

Note that the transit capacities will be scaled in a pretty non-linear way, such that multiplying through by the scale factor might not make much sense. Nothing you can really do about this though.

If this becomes regularly used or project critical please add to readme and an example config.

btw, the post processing module could be used to automate further processing, eg flows/veh capacity.

Fixin

return veh_capacity
else:
return 5.0

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

veh_type = self.resources['transit_vehicles'].veh_id_veh_type_map.get(vehicle_id, None)
return self.resources['transit_vehicles'].veh_type_capacity_map.get(veh_type, 5)

i would have written this as above, but on second thoughts, yours might actually be faster if it's predominantly cars so ignore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Fred, thank you so much for pointing this out as well as your comment regarding scaling. We had some further discussion on this and decided that car capacities are too heavily affected by scaling and won't be too relevant unless if something like ridesharing is investigated. We thought we would focus more on pt and pt crowding at this stage so I think I might add something very similar to the link passenger counts handler and make 'cars' an invalid mode.

"""
super().__init__(config=config, mode=mode, groupby_person_attribute=groupby_person_attribute, **kwargs)

self.groupby_person_attribute = groupby_person_attribute
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i doubt this attribute groupby will ever be used? But also I think it doesn't really cause significant overhead so may aswell leave in

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, I was thinking about this as well, I wasn't sure whether or not it should be kept in there. I will look into simplifying / removing it in the next draft :)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that there is not too much harm leaving it in, other than readability and maintainability.

I don't think there shoulld be any aweful consequences is removed other than a headache

standing_capacity = float(elem.xpath("capacity/standingRoom/@persons")[0])
else:
seated_capacity = float(elem.xpath("capacity/@seats")[0]) # new structure
standing_capacity = float(elem.xpath("capacity/@standingRoomInPersons")[0])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is fine for now, it doesn't get called many times so speed not a concern. I will keep an eye on this format to see if it is the new standard, but meanwhwile happy this supports both.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a result of a breaking change between versions 1 and 2 of the vehicles definition schema, as described in my comment on the issue raised by @elizabethc-arup.

We should check the entire schema and make sure Elara supports both versions, not just for this element, but for anything that has changed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should get some unit testing around the else clause - it's currently untested:

Screen Shot 2022-03-30 at 21 47 10

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screen Shot 2022-04-05 at 16 03 03

mfitz
mfitz previously approved these changes Mar 23, 2022
Copy link
Contributor

@mfitz mfitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The feature looks good and it's great to see unit tests.

There are a bunch of files in the diff that seem like maybe they should be in .gitignore - all of those binary files in tests/test_intermediate_data for example. I cannot see them being used anywhere - are they just an unimportant byproduct of running the tests? If so, we shouldn't be version controlling them. Similarly we have some files in tests/test_outputs and out/ - do these belong in the versioned repo? Some of them seem to have conflicts too:

Screen Shot 2022-03-23 at 03 10 36

Soon we will be open sourcing Elara, so now is a good time to tidy up.

# Car
# subpopulation breakdown
@pytest.fixture
def test_car_capacity_count_handler_with_subpopulations(test_config, input_manager):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a good idea to name a fixture test_, because that makes it look like a test, to both humans and the pytest test runner.

handler = test_car_capacity_count_handler_with_subpopulations
for elem in events:
handler.process_event(elem)
assert np.sum(handler.counts) == 70
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we expect it to be 70? Is this implicit knowledge about the fixture? Is there a way to be more explicit about our expectation?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Michael the 70 comes from the total capacity of the bus vehicle defined in the vehicles.xml file. I can't think of another way of embedding it to this test apart from importing and parsing the vehicles.xml, for now I have added a comment next to the assert stating where the number comes from, however, I am aware that this might not be the best practice. Would you be able to give me some advice on how I can best approach this? :(

handler.finalise()
assert len(handler.result_dfs) == 2
for name, gdf in handler.result_dfs.items():
cols = list(range(handler.config.time_periods))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should cols be defined outside of the loop?

assert c in gdf.columns
assert 'total' in gdf.columns
df = gdf.loc[:, cols]
assert np.sum(df.values) == 14*5 / handler.config.scale_factor
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, it's not clear why we have this expectation; 14 * 5 is a magic number here. Can we be clearer about why we expect what we expect?


# No class breakdown
@pytest.fixture
def test_car_capacity_count_handler_simple(test_config, input_manager):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps a better name would be something like car_link_vehicle_capacity_handler? Or maybe even something more descriptive that describes how this handler is configured?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you Michael, I have renamed all the test names to something that is a lot more descriptive as you suggested :)

@elizabethc-arup elizabethc-arup dismissed stale reviews from mfitz and fredshone via 3d9fe7f March 28, 2022 11:28
@elizabethc-arup elizabethc-arup force-pushed the extract-link-pt-capacity branch from 3d9fe7f to cad47d0 Compare March 28, 2022 11:50
@elizabethc-arup elizabethc-arup requested a review from mfitz March 28, 2022 11:56
mfitz
mfitz previously approved these changes Mar 30, 2022
Copy link
Contributor

@mfitz mfitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've made some minor changes to a couple of unit tests to remove some of the magic numbers and make the tests easier to understand, but we could still do a little more along those lines.

standing_capacity = float(elem.xpath("capacity/standingRoom/@persons")[0])
else:
seated_capacity = float(elem.xpath("capacity/@seats")[0]) # new structure
standing_capacity = float(elem.xpath("capacity/@standingRoomInPersons")[0])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should get some unit testing around the else clause - it's currently untested:

Screen Shot 2022-03-30 at 21 47 10

Copy link
Contributor

@mfitz mfitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the missing tests, so I think we are probably ready to go now @elizabethc-arup.

standing_capacity = float(elem.xpath("capacity/standingRoom/@persons")[0])
else:
seated_capacity = float(elem.xpath("capacity/@seats")[0]) # new structure
standing_capacity = float(elem.xpath("capacity/@standingRoomInPersons")[0])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screen Shot 2022-04-05 at 16 03 03

@@ -1,12 +1,14 @@
click==7.0
colorama==0.4.4
Fiona==1.8.20
Copy link
Contributor

@mfitz mfitz Apr 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I needed to add these pinned versions of a couple of transitive dependencies to make the unit test suite work in my local environment. May just have been something specific to my own setup, but I built the env from scratch and it was broken. Pinning to these versions hasn't broken the build, and it may well be a no-op for most people/environments.

@@ -503,3 +504,29 @@ def test_load_input_manager(test_xml_config, test_paths):
assert input_workstation.resources['plans']
assert input_workstation.resources['input_plans']
assert input_workstation.resources['mode_map']


def test_parses_vehicle_capacity_from_file_using_vehicle_definitions_1_schema():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are the tests for the if/else around the schema version in inputs.transform_veh_type_elem.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Michael, thank you so much. Creating independent version 1 and version 2 xmls and tests make a lot of sense and is a lot clearer and easier to follow than what I have been trying to implement. I will keep this in mind when I implement my next set of pytests, thank you once again :D

@elizabethc-arup elizabethc-arup merged commit 9ddc63e into main Apr 6, 2022
@mfitz mfitz deleted the extract-link-pt-capacity branch April 26, 2022 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants