NYPL Labs Building Inspector

This is the code repository for the Building Inspector. More information in the About page.

By: Mauricio Giraldo Arteaga / NYPL Labs

• Data ingest
• Consensus generation
• API querying
• Adding new tasks

Environment variables and buildpacks

Deploying in Heroku

UPDATE (2016/06/29): Heroku now properly handles multiple buildpacks natively.

This project makes use of several environment variables and buildpacks to work on Heroku. The RGeo gem in Heroku does not get built properly so some tasks do not work. You need to include these two buildpacks:

• https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git
• https://github.com/diowa/heroku-buildpack-rgeo-prep.git

You can do so by executing:

heroku buildpacks:add https://github.com/peterkeen/heroku-buildpack-vendorbinaries.git
heroku buildpacks:add https://github.com/diowa/heroku-buildpack-rgeo-prep.git

In order to verify that everything is working fine, enter the Heroku Rails console in the application and type:

RGeo::Geos.supported?

This should return => true. If this is not the case you may want to visit the issues list for heroku-geo-buildpack.

Social media login integration

This application makes use of OmniAuth which requires that you set up the necessary API keys for Google, Twitter and Facebook (links take you to the applicable area for each platform). Once procured, you need to set the following environment variables to their proper values:

• FACEBOOK_KEY
• FACEBOOK_SECRET
• GOOGLE_KEY
• GOOGLE_SECRET
• TWITTER_KEY
• TWITTER_SECRET

Google URIs

IMPORTANT: Make sure under "APIs" that you have the "Contacts API" and "Google+ API" enabled for this application.

In the APIs & auth > Credentials section of the Google Developers Console you need to create a Client ID for OAuth and set these values (replace {APPLICATION_URL} with the proper URL of the application):

• Redirect URIs: http://{APPLICATION_URL}/users/auth/google_oauth2/callback
• Javascript Origins: http://{APPLICATION_URL}

Data ingest

This project works with GeoJSON files generated by the NYPL Map Vectorizer. The specific files for the NYPL use case are found in the /public/files folder.

Read Importing Map Warper layers into Building Inspector for more detailed ingest instructions.

First ingest

After downloading, and running the proper rake db:migrate you need to do a base ingest of data using the included rake task.

All sheet data ingest

NOTE: You may need to prepend bundle exec to the Rake tasks if you're using Rails 4.

rake data_import:ingest_bulk id=LAYERNAME force=true

This assumes the presence of public/files/config-ingest-layerLAYERNAME.json with a list of IDs and bounding boxes to import for the layer LAYERNAME. It will create a layer whose external_id is the LAYERNAME provided. This erases all sheet/polygon/flag data for those IDs in the config file. If you don't have a config file, see the next section for instruction on how to build one easily.

Layer ingest config file creator

There is an included Python script -- ingestor_config_builder.py -- to easily create ingest config files to be used in the inspector. It assumes a folder full of GeoTIFF files (sheets) that you have vectorized and want to create in the database.

Usage:

python ingestor_config_builder.py /path/to/folder/with/geotiffs

It creates a config file in the application root folder with the name config-ingest-layerFOLDERNAME where FOLDERNAME is the name of the folder where the GeoTIFFs were found. In NYPL, the FOLDERNAME is the same as the LAYERNAME.

Single sheet data ingest

rake data_import:ingest_geojson id=SOMEID layer_id=SOMELAYERID bbox=SOMEBOUNDINGBOX force=true

This imports polygons from a file public/files/SOMEID-traced.json into the database replacing any polygons (and its corresponding flags) that are associated to a sheet with map_id equal to SOMEID. IMPORTANT: There has to be a layer whose id is equal to SOMELAYERID for the links to work! This is not the layer's external_id.

NOTE: (this only applies if you want to use NYPL polygons) So far only layers 859 and 860 are provided. Layer 859 has separate GeoJSON for centroids and polygons. Layer 860 sheets have a single file with both fields. Ingesting 859 requires a separate data_import:ingest_centroid_bulk process for centroids. See above script.

Centroids

NOTE: The vectorizer now adds centroids to the polygons vectorized. In case you have a GeoJSON file with polygons and no centroids, use this script below. Otherwise, you do not need these rake tasks.

Add bulk centroids

The original GeoJSON files do not have centroids (they were added and processed later). To create the centroids of the polygons in the database you need to run:

rake data_import:ingest_centroid_bulk id=LAYERNAME force=true

Single sheet centroid updating

This updates the polygon centroids for a given sheet_id from a file public/files/SOMEID-traced.json replacing any existing polygon centroids (not the polygons themselves):

rake data_import:ingest_centroids_for_sheet id=SOMEID force=true

Consensus generation

As people inspect polygons according to the task (eg. "YES/NO/FIX" for the geometry task) they are essentially casting a vote alongside fellow inspectors. We show the same polygon and task to several people and tally up those votes to decide whether they agree. If they do, that polygon is removed from the pool for that given task.

Each task has a different means to come to consensus. So far, consensus for geometry, color and basic address (value being NONE) are implemented. A rake task is available for this in lib/tasks/flag_processing.rake:

rake db:calculate_consensus

This task should be scheduled to execute regularly (say, every ten minutes). New consensus generation for other tasks is being implemented.

Calculating user rankings

The homepage of the site shows, for users who authenticate via Facebook, Google, or Twitter, their ranking in terms of amount of inspections. This is performed by the tasks in lib/tasks/user.rake:

rake user:calculate_scores

This task should be scheduled to execute regularly (every hour or so).

API querying

See a complete description in the Data page.

Adding new tasks

There is a very superficial README document briefly describing what changes need to be made to add new tasks to the Building Inspector.

Version notes

• 3.1 Now flags (votes) from users with role admin count as 4 votes. This is so that certain “power users” can do inspections that do not require additional views and reach consensus immediately.
• 3.0 Third release with toponym task. Refactored and improved code. New API endpoints.
• 2.0 Second release with address, polygonfix and color tasks. Refactored and improved code. New API endpoints.
• 1.1 Added API endpoints
• 1.0 First release with single geometry task

License notice

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.