Merge pull request #247 from pelias/docs

Move documentation from Pelias Wiki to this repo

docs/conflation.md

@@ -0,0 +1,143 @@
+## Conflation
+
+Pelias imports address data from many different sources; some contain line geometry while others contain address points or interpolation ranges.
+
+None of our source data sets currently contain street<>address concordances, grouping data is necessary in order to establish the data refers to the same entity/entities.
+
+It's important to be able to combine these data in order to:
+
+- increase our street address coverage
+- reduce 'holes' in the data which can cause a loss of precision
+- associate point data to road network data
+- deduplicate results
+
+This document outlines the different types of data we import and suggests some grouping techniques we can use to associate them.
+
+### Conflation for search
+
+Conflation for search is a similar process as for routing and display, the only major difference is that we aim to reduce the amount of duplicate street names to a minimum.
+
+Returning a list of osm ways is not acceptable for search as it will result in an experience such as:
+
+![conflation-problem](http://missinglink.embed.s3.amazonaws.com/search-conflation-problem.png)
+
+A better user experience would be to provide 'address ranges' of the street:
+
+![conflation-problem2](http://missinglink.embed.s3.amazonaws.com/search-conflation-problem2.png)
+
+An ideal experience would be to provide the exact street address:
+
+![conflation-problem3](http://missinglink.embed.s3.amazonaws.com/search-conflation-problem3.png)
+
+### Grouping Openstreetmap entities
+
+Conflation of openstreetmap entities is a well studied domain; both the routing team and the vector tiles team members have extensive experience in this area.
+
+#### Single line
+
+The most basic streets in Openstreetmap are an ordered collection of 'nodes' grouped together in a 'way', the street is given a `name` tag and a `highway:*` tag:
+
+| geometry | tags |
+|:-:|:-:|
+| ![osm-street-simple](http://missinglink.embed.s3.amazonaws.com/osm-street-simple.png) | ![osm-street-simple-tags](http://missinglink.embed.s3.amazonaws.com/osm-street-simple-tags.png) |
+
+These entities are relatively simple to import as they do not contain multiple line segments, some care will need to be taken when computing the centroid value; which should lie on the line string rather than in the center of the bounding-box.
+
+#### Multiple lines
+
+More complex roads require the road be split up in to 2 or more 'ways', such as this example, we have a road split in to 3 different 'ways'; two road segements with a roundabout in the middle:
+
+| geometry | tags |
+|:-:|:-:|
+|![osm-street-multiple-1](http://missinglink.embed.s3.amazonaws.com/osm-street-multiple-1.png) | ![osm-street-multiple-1-tags](http://missinglink.embed.s3.amazonaws.com/osm-street-multiple-1-tags.png) |
+|![osm-street-multiple-2](http://missinglink.embed.s3.amazonaws.com/osm-street-multiple-2.png) | ![osm-street-multiple-2-tags](http://missinglink.embed.s3.amazonaws.com/osm-street-multiple-2-tags.png) |
+|![osm-street-multiple-3](http://missinglink.embed.s3.amazonaws.com/osm-street-multiple-3.png) | ![osm-street-multiple-3-tags](http://missinglink.embed.s3.amazonaws.com/osm-street-multiple-3-tags.png) |
+
+These entities are usually related by common 'nodes', usually at the extremes of the constituent line segments.
+
+In some cases the line string may be 'broken', it may not share common nodes due to an obstacle such as an intersection or another feature.
+
+A spatial search can be performed to attempt to find other nodes in close spatial proximity which belong to a way which share the same name.
+
+Some caveats to avoid are:
+
+1. some cities have two or more roads with the same name, joining them would be incorrect.
+2. there is potential for different spelling between ways, a street name normalization function should be used to detemine if the names 'match' or not.
+
+Again, some care must be taken when computing a centroid value for the network. There is potential for the joined road network to be very long, it may span several different neighborhoods (or even states!).
+
+There will be cases where a single centroid value doesn't make sense. It might be best to split these entities in to 2 different road networks in order to provide a more intuitive search to the user.
+
+#### Disjoined lines
+
+The most complex case is when two parts of are road a broken up by large spatial gaps, this is fairly common in large cities where building development has divided existing streets.
+
+A good example is Golden Gate Park in San Francisco, here you can see that all the north/south avenues are completely disjoined by the park:
+
+![golden-gate](http://missinglink.embed.s3.amazonaws.com/golden-gate-park.png)
+
+New York city has adopted an convention for these disjoined streets, usually prefixing their names with either 'East' or 'West' in order to disambiguate the two sides of the park.
+
+Parks are only the 'tip of the iceberg' regarding disjoined road networks, there are very long and complex networks just as major highways to consider as well as cases where roads change names and then change back again.
+
+It's also common for road networks to be disjoined multiple times, such as in [this example](https://gist.github.com/missinglink/564835c5465bf83dac9056d77da9c529).
+
+There is much more complexity to this problem than covered here, again great care must be taken when computing a centroid value for these networks as:
+
+1. a bounding box centroid would result in the centroid being inside the obstacle rather than on the road network.
+2. the road network can be very large, spanning multiple cities/states or countries!
+
+![oresund-bridge](http://missinglink.embed.s3.amazonaws.com/oresund-bridge.png)
+
+#### Irregular geometries
+
+The world is a weird and wonderful place, it's best not to assume anything about how road networks are constructed, there will always be unusual geometries to be found, such as this:
+
+![earls-court-sq](http://missinglink.embed.s3.amazonaws.com/earls-court-square.png)
+
+It's best not to dwell on these unusual geometries as they are the exception rather than the rule.
+
+### Linking point data to the road network
+
+None of the address point data we source contain road network concordances, not even Openstreetmap!
+
+In the image above you can see that the house numbers are not positioned exactly on the road network itself. It's very uncommon to find a house which sits exactly on the street, there is usually a sidewalk or driveway which offsets the distance from the road network.
+
+#### Openstreetmap nodes
+
+For this reason, the OSM entity tagged with `addr:housenumber` rarely shares a common node with the road network; moreover the building way rarely shares a common node with the road network:
+
+| geometry | tags |
+|:-:|:-:|
+|![earls-court-51-node](http://missinglink.embed.s3.amazonaws.com/earls-court-51-node.png) | ![earls-court-51-node](http://missinglink.embed.s3.amazonaws.com/earls-court-51-node-tags.png) |
+|![earls-court-51-way](http://missinglink.embed.s3.amazonaws.com/earls-court-51-way.png) | ![earls-court-51-way](http://missinglink.embed.s3.amazonaws.com/earls-court-51-way-tags.png) |
+
+In order to group these street numbers with the road network to which they belong, we must use the same technique as discussed above:
+
+Using a combination of spatial distance and linguistic similarity we can; with a degree of confidence; establish to which road segment the street number belongs to.
+
+#### Openstreetmap ways
+
+Similar to the above; except in this case the `addr:housenumber` tag has been applied to the way itself rather than a node:
+
+| geometry | tags |
+|:-:|:-:|
+|![192-finborough-rd](http://missinglink.embed.s3.amazonaws.com/192-finborough-rd.png) | ![192-finborough-rd](http://missinglink.embed.s3.amazonaws.com/192-finborough-rd-tags.png) |
+
+#### Openstreetmap interpolation lines
+
+Openstreetmap contains invisible 'interpolation lines'; these ways group a range of addresses with a guide path which shows where the missing house numbers should lie:
+
+| geometry | tags |
+|:-:|:-:|
+|![wetherby-mansions-way](http://missinglink.embed.s3.amazonaws.com/wetherby-mansions-way.png) | ![wetherby-mansions-way](http://missinglink.embed.s3.amazonaws.com/wetherby-mansions-way-tags.png) |
+|![wetherby-mansions-node2](http://missinglink.embed.s3.amazonaws.com/wetherby-mansions-node2.png) | ![wetherby-mansions-node2](http://missinglink.embed.s3.amazonaws.com/wetherby-mansions-node2-tags.png) |
+|![wetherby-mansions-node1](http://missinglink.embed.s3.amazonaws.com/wetherby-mansions-node1.png) | ![wetherby-mansions-node1](http://missinglink.embed.s3.amazonaws.com/wetherby-mansions-node1-tags.png) |
+
+These lines can likely be processed after the nodes, if the nodes have already been associated to a road segment then that information can simply be copied to the interpolated address points.
+
+#### Openaddresses and other point-only address datasets:
+
+Other datasets which only contain point data can use the same process to create concordances between their house numbers and the road network:
+
+![os-extract.png](http://missinglink.embed.s3.amazonaws.com/oa-extract.png)

docs/design-doc.md

@@ -0,0 +1,118 @@
+
+## Interpolation
+
+This document outlines a proposal for refactoring how street addresses are stored and retrieved in Pelias.
+
+> Se also: [An introduction to addresses in Pelias](introduction.md)
+
+The strategic goals of the work are:
+
+- Ensuring every street in Openstreetmap is indexed and retrievable.
+- Supporting address ranges as provided by Openstreetmap, TIGER, et al.
+- Combining and de-duplicating distinct address point data sets.
+- Designing the system to scale beyond 1B address points.
+- Allow room for future extension / improvements.
+
+These changes will allow for the following user experience improvements:
+
+- Provide house number interpolation where address range data exists.
+- Fall back to providing a street centroid in lieu of a satisfactory house number.
+- Reduce noise by only showing a maximum of one result per street.
+
+### Source data
+
+The work requires a conflated `road network` dataset and one or more `house number` datasets in order to function. Additional house number data sets will improve the coverage and accuracy of the system.
+
+> See also: [The problem of conflation outlined in more detail](conflation.md)
+
+### Road network
+
+It is essential to have a pre-processed and conflated road network in order to:
+
+- Reduce / avoid duplicate street names in results.
+- Provide line strings which can act as interpolation guides.
+- Ensure that interpolated points do not lie in a driving hazard.
+
+Three strategies for conflating the OSM road network were considered:
+
+- Create a new system to conflate the OSM ways *- too time consuming and error prone*.
+- Extract data from vector tiles *- not appropriate due to file size optimizations and entity merging*.
+- Utilize the routing graph *- similar domain, not concerned with any entities except roads*.
+
+#### Exporting the Valhalla routing graph
+
+![generic graph image](http://i.stack.imgur.com/JrBdQ.png)
+
+Valhalla doesn't store OSM ways, it breaks up the source data in to a graph of 'edges'. Each edge is marked up with [meta data such as this](https://gist.github.com/missinglink/b2ac67f51d132b591868a9ef60061c43).
+
+By [iterating over all the tiles](https://github.com/valhalla/tools/issues/60) we can walk the graph and join adjacent road segments with the same name.
+
+The process would take around 2 hours and would produce a dump file containing:
+
+- a single continuous line string representing the geometry of the road
+- the street name
+- (optionally) meta data about the street such as direction
+- (optionally) a centroid value for the street
+
+**note:** the algorithm will favour the longest contiguous path, in the case of disjoined streets and geometries that cannot be represented using a single line; a second line will be produced with the same street name.
+
+Future work can be planned in v2 to:
+
+- Improve the name matching algorithm.
+- Reduce the number of duplicate line segments produced.
+- Break line strings on geographic / political boundaries.
+
+### Point data
+
+![generic house number image](http://wiki.openstreetmap.org/w/images/f/f2/Housenumber_example_kms_2.png)
+
+Point data from Openstreetmap and OpenAddresses will need to be associated to the correct segment of the road network (a single entry from the dump above).
+
+Given only a lat/lon pair and a street name, the system must be able to quickly find the appropriate road network segment and retrieve a unique ID representing it.
+
+As above, the quality of the street normalization and spelling error detection will affect the quality of the matching algorithm.
+
+The house number point should then be [projected on the line string](http://stackoverflow.com/questions/10301001/perpendicular-on-a-line-segment-from-a-given-point), this will give us a new point which is guaranteed to lie on the line string.
+
+The projected point data is saved along with the original position, one will be used for exact matches while the other will provide interpolation data.
+
+### Range data
+
+![range](http://missinglink.embed.s3.amazonaws.com/osm-interpolation-tag.png)
+
+Range data from Openstreetmap and TIGER will also need to be associated with the correct segment of the road network.
+
+> See also: [Information about existing interpolation range standards](existing-standards.md)
+
+There is a performance vs. index size tradeoff that can be made here, either the range data can be 'expanded' at index time or at query time. It seems that query time expansion of ranges would be preferable as it keeps the index smaller and allows behavioral modifications without a full reindex.
+
+Judging by the tag statistics in the link above we will get much more value from supporting `addr:interpolation:*` tags than TIGER `from_address_right` etc tags. I would recommend not supporting TIGER tags in the v1 work.
+
+The OSM interpolation tags simply join two of the points mentioned above, so importing these ranges is very easy, simply associate them to the same road network as the child points, no further projection need be performed at this time.
+
+### Importing in to Pelias
+
+Pelias will require an import of one document per line string in the road network, this will be in the ten-of-millions.
+
+This data should be imported in to a new layer, named `street` which distinguishes it from `address` and `venue` data.
+
+Each record will require a centroid, if one is not provided in the source data then it will need to be computed.
+
+The line strings should not be stored in Elasticsearch, they have the potential to be very large (10's of GB).
+
+### Query logic
+
+```
+Can we determine candidate street(s) based on the input text?
+[no] Fail
+
+Did the user provide a house number in the query?
+[no] Return the street centroid
+
+Do we have point data for the requested house number?
+[yes] Return the exact position
+
+Do we have an address range encompassing the requested house number?
+[yes] Return the interpolated position
+[no] Return the street centroid
+```