Merge pull request #69 from tooledesign/dev

Version 1.0.0 release!

README.md

@@ -22,7 +22,8 @@ pyBNA is tested with Python 3.6. The following libraries are required:
 - geopandas
 - munch
 - overpass
-- omsnx
+- osmnx
+- xlrd
 
 A requirements.txt file is provided for convenience. You can install these via
 pip:
@@ -64,7 +65,10 @@ s.crossing_stress()
 # connectivity
 bna = pybna.pyBNA()
 bna.calculate_connectivity()
-bna.score_destinations("myschema.mytable")
+
+# scores
+bna.score("myschema.my_scores_table")
+bna.aggregate("myschema.my_aggregate_score_table")
 ```
 
 ## Importing data
@@ -76,6 +80,15 @@ OpenStreetMap.
 
 For more guidance on the import process, see our [import instructions](import.md).
 
+## Traffic Stress
+
+pyBNA has a module that can calculate traffic stress based on roadway
+characteristics in your roadway data. There's also the ability to apply
+assumptions for locations where data is not available.
+
+For more information about the traffic stress module, see the [traffic stress
+instructions](stress.md).
+
 ## Getting started
 
 First, import pybna and create a pyBNA object by pointing it to the config file.
@@ -91,7 +104,12 @@ bna.calculate_connectivity()
 
 Lastly, you can generate block-level scores with
 ```
-b.score_destinations("my_results_table")
+bna.score("myschema.my_scores_table")
+```
+
+and aggregate scores for the entire study area with
+```
+bna.aggregate("myschema.my_aggregate_score_table")
 ```
 
 ## Configuration file
@@ -108,3 +126,7 @@ Once you've completed the connectivity analysis, you can develop a low/high stre
 ```
 bna.travel_sheds([list, of, block, ids, here], my_travel_shed_table)
 ```
+
+## Scenarios
+
+pyBNA includes the capability to run scenarios as a way to visualize the connectivity impacts of a project or group of projects. More information on scenarios is available [here](scenarios.md).

config.md

@@ -4,7 +4,16 @@ Most of the flexibility in pyBNA is managed using a configuration file. This
 file is passed to the pyBNA object as an argument when it is created and tells
 pyBNA important things about the input data and assumptions used in the process. An example configuration file using standard BNA defaults can be found [here](pybna/config.yaml).
 
-The first section tells pyBNA how to connect to the database
+`srid` identifies the map projection to use for the data
+
+`units` identifies whether speed limits and widths in the road data are given in
+imperial (`mi`) or metric (`km`) units. Note that spatial numbers given
+elsewhere in the configuration file are in the units associated with the `srid`.
+This means, for example, you could be using imperial units (mph and feet) in your
+roadway data, but the distance for `max_distance` could be given in meters if
+your projection uses meters.
+
+The first grouped section tells pyBNA how to connect to the database
 ```
 db:
     user: "gis"
@@ -35,7 +44,7 @@ geom | Name of the geometry column |
 ```
     blocks:
         table: "generated.neighborhood_census_blocks"
-        id_column: "blockid10"
+        uid: "blockid10"
         population: "pop10"
         geom: "geom"
         roads_tolerance: 15
@@ -47,7 +56,7 @@ This section tells pyBNA about the table of population areas used as the unit of
 Entry | Description | Required
 :--- | :--- | :---:
 table | Name of the table | X
-id_column | Name of the table's primary key |
+uid | Name of the table's primary key |
 population | Name of the population attribute | X
 geom | Name of the geometry column |
 roads_tolerance | Tolerance used when searching for roads that are associated with a block | X
@@ -75,16 +84,17 @@ min_road_length | Length a roadway must share with a block area in order to be c
             table: "neighborhood_ways_intersections"
             geom: "geom"
             uid: "int_id"
+            cluster_distance: 1
         edges:
             table: "neighborhood_ways_net_link"
             source_column: "source_vert"
             target_column: "target_vert"
             stress_column: "link_stress"
             cost_column: "link_cost"
-            id_column: link_id
+            uid: link_id
         nodes:
             table: "neighborhood_ways_net_vert"
-            id_column: vert_id
+            uid: vert_id
 ```
 
 
@@ -108,6 +118,9 @@ name | Name of the attribute holding one way designation | X
 forward | Value indicating one way in the forward direction | X
 backward | Value indicating one way in the backward direction | X
 
+N.B. If there are values in the one-way column that don't match either the
+forward or backward value the road is considered to be two-way.
+
 `segment stress`
 
 Entry | Description | Required
@@ -129,6 +142,7 @@ Entry | Description | Required
 table | Name of the table | X
 geom | Name of the geometry column |
 uid | Primary key |
+cluster_distance | Tolerance for grouping nearby vertices | X
 
 #### edges
 
@@ -139,14 +153,14 @@ source_column | Name of the attribute indicating the source node | X
 target_column | Name of the attribute indicating the target node | X
 stress_column | Name of the attribute indicating the LTS on the edge | X
 cost_column | Name of the attribute indicating the cost of the edge | X
-id_column | Primary key | X
+uid | Primary key | X
 
 #### nodes
 
 Entry | Description | Required
 :--- | :--- | :---:
 table | Name of the table | X
-id_column | Primary key | X
+uid | Primary key | X
 
 ### connectivity
 
@@ -160,7 +174,8 @@ table | Name of the table | X
 source_column | Name of the attribute holding the ID of the source block | X
 target_column | Name of the attribute holding the ID of the target block | X
 max_distance | The maximum distance to search for possible block connections | X
-max_detour | The maximum percentage to exceed high-stress distance and still be considered connected on the low-stress network (given is a whole number) | X
+max_detour | The maximum percentage to exceed high-stress distance and still be considered connected on the low-stress network (given as a whole number out of 100) | X
+detour_agnostic_threshold | Distance under which the % detour is ignored. As long as a low-stress connection is under this threshold it is counted even if it is significantly longer than the high-stress alternative | X
 max_stress | The maximum LTS score to consider for low-stress connectivity | X
 
 ### destinations
@@ -189,16 +204,43 @@ Entry | Description | Required
 name | Category name | X
 weight | Category weight | X
 table | Name of the table | X
-method | Either "count" or "percentage" | X
-datafield | Attribute used to calculate percentage<br>(only used for "percentage" method) | depends
+method | Either `count` or `percentage` | X
+datafield | Attribute used to calculate percentage<br>(only used for `percentage` method) | depends
 maxpoints | Total possible points for this category | X
 breaks | List of breaks at which points are awarded | X
-uid | Primary key |
-blocks | Attribute with blocks associated with each destination | X
+uid | Primary key<br>(for `percentage` method this must match the primary key of the census blocks table) |
+geom | Name of the geometry column |
+osm_tags_query | See below for additional documentation |
 
 The category weight is relative to its peers. In other words, the weights of
 subcategories are unrelated to the weight of their parent categories.
 
-Breaks are defined values at which the specified number of points are awarded. Any results that sit between the given breaks are pro-rated based on the surrounding break values.
+Breaks are defined values at which the specified number of points are awarded.
+Any results that sit between the given breaks are pro-rated based on the
+surrounding break values.
+
+The `percentage` method assigns points cumulatively based on the ratio of the
+given attribute within low-stress access divided by the high-stress access. For
+example, if a block has high-stress access to 100 people and low-stress access
+to only 30 people, its percentage is 0.3 or 30%.
+
+`osm_tags_query` allows you to define an OSM query for extracting these
+destinations directly from OSM. The query follows the [Overpass
+API](https://wiki.openstreetmap.org/wiki/Overpass_API) format. Queries can be
+given as a list to combine multiple queries. For example, the query for doctor
+offices would use the following format:
+
+```
+osm_tags_query:
+    - "['amenity'='doctors']"
+    - "['amenity'='doctor']"
+    - "['amenity'='clinic']"
+```
+
+The queries are applied as part of the `import_osm_destinations` method in
+pybna's [Importer](import.md#Destinations) class.
+
+### stress
 
-The percentage method assigns points cumulatively based on the ratio of the given attribute within low-stress access divided by the high-stress access. For example, if a block has high-stress access to 100 people and low-stress access to only 30 people, its percentage is 0.3 or 30%.
+Information about the stress portion of the configuration file is provided in
+the `stress configuration` section of the [stress instructions](stress.md).

import.md

@@ -160,13 +160,22 @@ given in your config file.
 If you already downloaded an OSM extract locally you can refer to it instead of
 pulling data over the network with the `osm_file` option.
 
+**Note** that very large OSM files can result in memory errors (computer running
+out of memory). One strategy to avoid this would be to isolate only features
+with the tag `highway=*`. There are several options for filtering raw OSM data,
+such as Osmium:
+```
+osmium tags-filter -o ~/path/to/output/file ~/path/to/input/file nw/highway
+```
+
 # Destinations
 
 Destination data also comes from OpenStreetMap by default. The Importer uses the
-default destination categories and definitions for the BNA, but you can provide
-your own instructions for extracting OSM destinations using the
-`destination_tags` option. To do this, you'll need to create a dictionary of
-table names and OSM tags that mimics the default baked into the code.
+destination tags defined in the [configuration file](config.md#destinations),
+but you can provide separate instructions for extracting OSM destinations using
+the `destination_tags` option. To do this, you'll need to create a dictionary of
+table names and OSM tags that mimics the format encoded in the configuration
+file.
 
 Importing destinations can be done with:
 ```

pybna/__init__.py

@@ -1,3 +1,4 @@
 from .pybna import pyBNA
 from .importer import Importer
 from .stress import Stress
+from .tests import *

pybna/conf.py

@@ -11,7 +11,7 @@
 
 
 class Conf(DBUtils):
-    """pyBNA Connectivity class"""
+    """pyBNA configuration class"""
 
     def __init__(self):
         DBUtils.__init__(self,"")
@@ -23,8 +23,10 @@ def parse_config(self,config):
         Reads through the giant dictionary loaded from YAML and converts into
         munches that can be accessed with dot-notation
 
-        args:
-        config -- a dictionary of configuration options
+        Parameters
+        ----------
+        config : dict
+            a dictionary of configuration options
 
         returns:
         Munch
@@ -226,8 +228,10 @@ def _build_segment_sql_substitutions(self,direction):
         Builds commonly-shared segment-oriented SQL substitutions from the
         entries in the config file
 
-        args:
-        direction -- the direction to generate substitutions for
+        Parameters
+        ----------
+        direction : str
+            the direction to generate substitutions for
 
         returns:
         a dictionary holding SQL objects
@@ -266,6 +270,19 @@ def _build_segment_sql_substitutions(self,direction):
         else:
             assumed_centerline = sql.SQL("FALSE")
 
+        # low_parking
+        if "low_parking" in settings:
+            low_parking_column = sql.Identifier(settings["low_parking"]["name"])
+            low_parking_value = sql.Literal(settings["low_parking"]["val"])
+        else:
+            low_parking_column = sql.SQL("NULL")
+            low_parking_value = sql.SQL("NULL")
+        low_parking = sql.SQL("({}={})").format(low_parking_column,low_parking_value)
+        if "low_parking" in assumptions:
+            assumed_low_parking = self._build_case(assumptions["low_parking"])
+        else:
+            assumed_low_parking = sql.SQL("FALSE")
+
         # speed
         if "speed" in settings:
             speed = sql.Identifier(settings["speed"])
@@ -276,6 +293,16 @@ def _build_segment_sql_substitutions(self,direction):
         else:
             assumed_speed = sql.SQL("NULL")
 
+        # width
+        if "width" in settings:
+            width = sql.Identifier(settings["width"])
+        else:
+            width = sql.SQL("NULL")
+        if "width" in assumptions:
+            assumed_width = self._build_case(assumptions["width"])
+        else:
+            assumed_width = sql.SQL("NULL")
+
         # oneway
         if "oneway" in settings:
             oneway_column = sql.Identifier(settings["oneway"]["name"])
@@ -395,8 +422,12 @@ def _build_segment_sql_substitutions(self,direction):
             "assumed_lanes": assumed_lanes,
             "centerline": centerline,
             "assumed_centerline": assumed_centerline,
+            "low_parking": low_parking,
+            "assumed_low_parking": assumed_low_parking,
             "speed": speed,
             "assumed_speed": assumed_speed,
+            "width": width,
+            "assumed_width": assumed_width,
             "aadt": aadt,
             "assumed_aadt": assumed_aadt,
             "parking": parking,
@@ -428,8 +459,10 @@ def _build_crossing_sql_substitutions(self,direction):
         Builds crossing SQL substitutions from the entries in the config
         file
 
-        args:
-        direction -- the direction to generate substitutions for
+        Parameters
+        ----------
+        direction : str
+            the direction to generate substitutions for
 
         returns:
         a dictionary holding SQL objects
@@ -657,3 +690,43 @@ def _build_case(self,vals,prefix=None):
             case += sql.SQL(" ELSE ") + sql.Literal(vals[-1]["else"])
         case += sql.SQL(" END ")
         return case
+
+
+    def get_destination_tags(self):
+        """
+        Compiles a list of dictionaries that describe the tables and OSM tags
+        for destinations in the config file.
+
+        returns:
+        a list of dictionaries
+        """
+        tags = []
+        for destination in self.config.bna.destinations:
+            tags.extend(self._get_destination_tags(destination))
+        return tags
+
+
+    def _get_destination_tags(self,node):
+        """
+        Helper method to be used in recursing destinations for OSM tags.
+        Returns a list of dictionaries with the table and OSM tags for the given
+        node in the destinations. If the node has subcats, appends these to the
+        result.
+
+        Parameters
+        ----------
+        node : dict
+            A destination type in the config file
+
+        returns:
+        a list of dictionaries
+        """
+        tags = []
+        if "subcats" in node:
+            for destination in node["subcats"]:
+                tags.extend(self._get_destination_tags(destination))
+
+        if "table" in node and "osm_tags_query" in node:
+            tags.append({"table":node["table"],"tags_query":node["osm_tags_query"]})
+
+        return tags