Code

bulletproof#139

Code

Open

eads wants to merge 171 commits intomasterpropublica/il-ticket-loader:masterfrom 0127-bulletproofpropublica/il-ticket-loader:0127-bulletproofCopy head branch name to clipboard

+397-7,044

Commits

Commits on Apr 12, 2018

• initial loader

  eads

  committed
  9d79555

Commits on Apr 24, 2018

• initial commit

  eads

  committed
  5323d84

Commits on Jun 11, 2018

• WIP makefile + benchmarks

  eads

  committed
  3449dc5

Commits on Jun 13, 2018

• add some git keeps

  eads

  committed
  0fb037a

• improve makefile and add camera data loader

  eads

  committed
  13a2933

• add sql definitions for parking and cameras, though the schema is identical

  eads

  committed
  d674643

• add csv cleaning processor

  eads

  committed
  f94b7e6

• add dev env var file

  eads

  committed
  58490be

• remove duplicate recipe and consolidate aliases

  eads

  committed
  4a04340

• update readme to account for consolidated aliases and some broken parts

  eads

  committed
  9759acc

Commits on Jul 6, 2018

• add clean commands

  eads

  committed
  1c690cb

• move up download command

  eads

  committed
  4816597

• fix circular reference

  eads

  committed
  d6a6e3e

• fix typo in download rule

  eads

  committed
  f2eb5d7

• download command for cameras

  eads

  committed
  a47a428

Commits on Jul 9, 2018

• change error file from .csv to .txt, closes #8

  

  kat-alo

  committed
  d6cb835

• change psql table name from 'tickets' to 'parking', closes #10

  

  kat-alo

  committed
  774d81f

Commits on Jul 11, 2018

• add skeletal file

  eads

  committed
  a032a23

• factor out cleaning functions

  eads

  committed
  25ed78c

• simple offset-based geocoder script, plus related schema updates

  eads

  committed
  83f0d84

• Add geocodes table

  eads

  committed
  9a07337

• add geocode tale maker to makefile

  eads

  committed
  4620e37

• error trapping, insert into unique db, and more fields in geocode insertion

  eads

  committed
  62950cb

Commits on Jul 12, 2018

• use ranges, not offsets and limits, in geocoder.

  Show description for 3e331e1
  eads

  committed
  3e331e1

Commits on Jul 13, 2018

• add some views

  eads

  committed
  b4b4d5a

• massive makefile overhaul

  Show description for bec3bc2
  eads

  committed
  bec3bc2

• match addresses on ID when inserting, better guardrails on grabbing elements of geocoded data

  Show description for 6adaf65
  eads

  committed
  6adaf65

• keep the dumps directory

  eads

  committed
  c4c1558

• table definition for community area stats from CMAP

  eads

  committed
  4ac288f

• add amusingly embarassing note to readme

  eads

  committed
  5447790

• Merge pull request #17 from propublica/0007-eads-geocoder

  Show description for 3f21496
  eads

  authored
  3f21496

Commits on Jul 16, 2018

• corrects = and :, cuts drop_db dependency on create_db, corrects reference to URL when DB_NAME needed, closes #22

  

  kat-alo

  committed
  018cbc4

• adds csvkit requirement to README closes #23

  

  kat-alo

  committed
  94d7e21

Commits on Jul 18, 2018

• replicate katlyn's makefile fixes

  eads

  committed
  8bd83a2

• correct dependency in load_community_areas make command

  eads

  committed
  aff7213

• fix community area matching, closes #25

  Show description for f482c25
  eads

  committed
  f482c25

• Merge pull request #27 from propublica/0025-community-area-matching

  Show description for 9f681fc
  eads

  authored
  9f681fc

Commits on Jul 19, 2018

• Merge branch 'master' of github.com:propublica/il-ticket-loader

  

  kat-alo

  committed
  469fa3b

Commits on Jul 20, 2018

• load dump file

  eads

  committed
  55fb3d9

• Merge branch '0025-community-area-matching'

  eads

  committed
  c7d0a8d

• ignore permissions from dump file, closes #30

  eads

  committed
  fabde92

• Merge pull request #32 from propublica/0030-pgrestore-perms

  Show description for f8f23bf
  eads

  authored
  f8f23bf

• add db string to environment

  eads

  committed
  9f4498f

• Merge pull request #33 from propublica/0031-dbstring

  Show description for 1878e72
  eads

  authored
  1878e72

Commits on Jul 23, 2018

• refactor make top level command signatures

  Show description for a9a5c98
  eads

  committed
  a9a5c98

• remove csvsql schema generation from makefile and turn into a note in the readme3

  eads

  committed
  08d9f69

• clean up restore permissions, change filename for now

  eads

  committed
  e8aa465

• remove irrelevant notes file

  eads

  committed
  af0ff11

• flip order of explanation about config

  eads

  committed
  fe65910

• reflect latest make targets

  eads

  committed
  7e9aac2

• handle 00s, closes #38

  eads

  committed
  1ec3af3

• fix db url reference

  eads

  committed
  dc2e482

• fix bad bug with wrong types for geocoder ranges

  eads

  committed
  07fd94d

• add quotes to community_area_stats table

  eads

  committed
  5bd3ece

• add geom column that's been giving us the flux

  eads

  committed
  328049a

• include just city sticker accuracy in geocode accuracy view

  eads

  committed
  46683fa

• add analysis view that joins every geocode with its community area

  eads

  committed
  8a91b61

• Merge branch 'master' of github.com:propublica/il-ticket-loader

  

  kat-alo

  committed
  76ff39c

Commits on Jul 25, 2018

• allows psql view to be updated when view is updated

  

  kat-alo

  committed
  b75b2a1

• adds median income to community area stats csv

  

  kat-alo

  committed
  b41e4a6

Commits on Aug 21, 2018

• adds all years to data and salts n hashes the license plate numbers, closes #42, #43

  

  kat-alo

  committed
  0bae864

Commits on Aug 22, 2018

• corrects naming of salt.txt file for the intermediate declaration I realized it was wrong on the bus my b

  

  kat-alo

  committed
  50cb16d

• Merge pull request #46 from propublica/0042-hash-plates

  Show description for 934ae71
  

  Kat-Alo

  authored
  934ae71

• exports parking table as one big csv and zips file, closes #50, closes #49

  

  kat-alo

  committed
  dedcd07

• zip_n_ship makefile command now also exports zip to s3

  

  kat-alo

  committed
  ceeafe6

• forget to add the salt creator my b

  

  kat-alo

  committed
  a5ecadc

• adds unit_key.csv to zip file to upload to s3

  

  kat-alo

  committed
  55d484d

• adding unit_key.csv to data/ in case Nicolas Cage needs it

  

  kat-alo

  committed
  2f64ebd

Commits on Aug 23, 2018

• Merge pull request #55 from propublica/0048-zip-n-ship-data

  Show description for f16b11e
  eads

  authored
  f16b11e

• Merge pull request #54 from propublica/0042-hash-plates

  Show description for e98910d
  eads

  authored
  e98910d

• adds data dict to zip file for s3

  

  kat-alo

  committed
  36e35e2

• adding requirements text file

  

  kat-alo

  committed
  19d0577

• Merge branch 'master' of github.com:propublica/il-ticket-loader

  

  kat-alo

  committed
  5d244de

• updating gitignore, closes #58

  

  kat-alo

  committed
  aa1b7b1

• updating readme to include data dictionary and python requirements, closes #52

  

  kat-alo

  committed
  5c57664

Commits on Oct 12, 2018

• account for reason for dismissal from older data

  eads

  committed
  b3d54ef

• makefile cleanup

  eads

  committed
  aed0720

• add dismissal reason to schema

  eads

  committed
  90d88b8

• block summary 'views' (really derived tables until hasura supports materialized views)

  eads

  committed
  2c071b7

• fix view creation order

  eads

  committed
  25656dc

• update readme with additional reqs, view order

  eads

  committed
  e062fb3

Commits on Oct 18, 2018

• first tests. Found one that's not passing, too

  eads

  committed
  b3186b3

Commits on Oct 19, 2018

• trying out circleci

  eads

  committed
  4a7440d

• rename clean_csv test file for consistency

  eads

  committed
  709abcf

• add test for clean_row function

  eads

  committed
  0979214

• use a config.yml that was autogenerated during project setup

  eads

  committed
  638bae7

• match zero or more leading zeros from address strings

  eads

  committed
  0882b44

• fix requirements for circle ci

  eads

  committed
  08704d0

• Merge pull request #71 from propublica/0070-leading-zeroes

  Show description for 52cbe31
  eads

  authored
  52cbe31

• Merge pull request #59 from propublica/0048-zip-n-ship-data

  Show description for 7ae0e2a
  eads

  authored
  7ae0e2a

Commits on Oct 22, 2018

• derive geocodes from original table

  eads

  committed
  71e9744

• try to speed up intermediate table query

  eads

  committed
  eb9d834

Commits on Oct 23, 2018

• more makefile optimization

  eads

  committed
  d86dacd

• make hash object once (cuts a few seconds off processing time)

  eads

  committed
  a3b141b

• use new year column in queries

  eads

  committed
  83ef995

• commit index creating to sql files

  eads

  committed
  7cc4869

• move test file back where it belongs

  eads

  committed
  230cc5f

• Merge pull request #76 from propublica/0074-dedupe-geocodes

  Show description for bf476ac
  eads

  authored
  bf476ac

Commits on Oct 24, 2018

• use C collation

  Show description for 7808e0b
  eads

  committed
  7808e0b

• join through multiple addresses and lat / lngs

  eads

  committed
  1211fcc

• the amount of branches with this fix is embarassing

  eads

  committed
  2152bf8

• Merge branch 'master' into 0078-extraction-tests

  eads

  committed
  dd6c67b

• fix mistake in hashing

  eads

  committed
  8e7611d

• clean up clean_csv file and factor date extractors into wee functions

  eads

  committed
  9ccfef6

• functional extraction tests

  eads

  committed
  8dba7e9

• Merge pull request #80 from propublica/0078-extraction-tests

  Show description for feb1f01
  eads

  authored
  feb1f01

• sane checks on existence of geocode tables and views

  eads

  committed
  daaec3e

• introduce geocodes_normalized view

  eads

  committed
  cfbbdef

• Merge pull request #84 from propublica/0077-partial-build-geo

  Show description for 91a9a72
  eads

  authored
  91a9a72

• Merge branch 'master' into 0075-better-indexing

  eads

  committed
  8490740

• used normalized table consistently

  eads

  committed
  171cf08

• include geom in normalized geocode query

  eads

  committed
  daddd0d

• Merge pull request #79 from propublica/0075-better-indexing

  Show description for 577c675
  eads

  authored
  577c675

• write files to dupes dir, closes #88

  eads

  committed
  f2c273f

• Merge pull request #91 from propublica/0088-fix-dupes

  Show description for 1ba2fbd
  eads

  authored
  1ba2fbd

• clean, correct, and add query example to readme

  Show description for 3bed0a4
  eads

  committed
  3bed0a4

• Merge pull request #94 from propublica/0089-readme-cleanup

  Show description for 553c94d
  eads

  authored
  553c94d

• fix bad data dict list in readme

  eads

  committed
  a378d52

Commits on Oct 25, 2018

• add violation lookup

  eads

  committed
  b73b131

• Merge pull request #96 from propublica/0092-violation-lookup

  Show description for 615be24
  eads

  authored
  615be24

Commits on Oct 29, 2018

• wip

  eads

  committed
  e0a87e6

• new core DB structure for ward analysis

  eads

  committed
  ba35118

• Merge pull request #97 from propublica/0081-ward-analysis-work-backward

  Show description for c494a93
  eads

  authored
  c494a93

• clean up, skip clustering for now

  eads

  committed
  1facd58

• Merge pull request #100 from propublica/0092-violation-lookup

  Show description for 7d6b7e9
  eads

  authored
  7d6b7e9

• load metadata

  eads

  committed
  430e29d

• Merge pull request #101 from propublica/0099-ward-meta

  Show description for d7ef46b
  eads

  authored
  d7ef46b

Commits on Nov 5, 2018

• calculate penalties; refactor data structure

  eads

  committed
  ca552da

• Merge pull request #108 from propublica/0105-calculate-penalties

  Show description for 4244acc
  eads

  authored
  4244acc

• ignore mapbox

  eads

  committed
  b6fad26

• restore parking

  eads

  committed
  8e2ae66

• Merge pull request #111 from propublica/0110-parking

  Show description for a3b3e22
  eads

  authored
  a3b3e22

Commits on Nov 7, 2018

• upload data to mapbox

  eads

  committed
  892e71c

• refactor penalty calculator a bit

  eads

  committed
  683e2bb

• move hasura over here

  eads

  committed
  e6ad632

• Merge pull request #112 from propublica/0098-hasura

  Show description for 23f714e
  eads

  authored
  23f714e

Commits on Nov 28, 2018

• make geo layers for app

  eads

  committed
  4b37297

• calculate % successfully contested

  eads

  committed
  4426bf3

Commits on Nov 29, 2018

• add make command to populate geocode table

  eads

  committed
  07aa9e6

• clean up makefile

  eads

  committed
  61bb0a4

• update db schema and indexes for geocodio

  eads

  committed
  bc8e146

• add scrapy geocoder

  eads

  committed
  f6ce55e

Commits on Nov 30, 2018

• functional new geocoder

  eads

  committed
  6f17159

• remove old geocoder

  eads

  committed
  3118910

• start loading census data

  eads

  committed
  0e24589

Commits on Dec 6, 2018

• wip, functional new geocoding

  eads

  committed
  6b4b389

Commits on Dec 10, 2018

• tweaks for app, tests

  eads

  committed
  119885b

• test % of debt captured in wards vs city wide figures

  eads

  committed
  f77b8eb

Commits on Dec 11, 2018

• use lte and gte for totals query

  eads

  committed
  a0846cd

• better block query

  eads

  committed
  fa1e498

Commits on Dec 12, 2018

• preliminary bulletproofing -- to be pipelined once we lock down details

  j2kao

  committed
  f15946c

Commits on Dec 13, 2018

• lots of small updates and fixes

  eads

  committed
  3d24b4f

• Merge branch 'master' into 0127-bulletproof

  eads

  committed
  cabc124

Commits on Dec 17, 2018

• tons of small data fixes

  eads

  committed
  dba762c

• bump to latest metadata

  eads

  committed
  f05a016

Commits on Dec 18, 2018

• check top 5 violation categories for 2013-2017

  j2kao

  committed
  49b7a46

• add top 5 and top level summaries

  j2kao

  committed
  8cb830c

Commits on Dec 20, 2018

• many data fixes for production

  eads

  committed
  d589fd9

Commits on Feb 12, 2019

• fix camera loader to accomodate older data

  eads

  committed
  2749c64

• do the pipfile thing

  eads

  committed
  69d3023

• camera data now runs through python processor and needs new columns

  eads

  committed
  7aa1029

• remove duplicate issue date in table definition

  eads

  committed
  b80f589

• add some exports for block club

  eads

  committed
  5694556

• ignore bulletproofing stuff

  eads

  committed
  fc58842

• make datastore directory

  eads

  committed
  27c31dc

• use exports framework to zip-n-ship

  eads

  committed
  7e69e02

• add makefile download alias

  eads

  committed
  7c7087c

• update README for release

  eads

  committed
  2452f30

• add exports and turn off s3 uploading for now

  eads

  committed
  7428b11

Commits on Mar 4, 2019

• add bulletproofing stuff to messy gitignore

  eads

  committed
  85ac89a

• merge in way that should allow for merging back to master

  eads

  committed
  7c21419