Back fill the Raw Inmate Data to August 17th, 2013

[nwinklareth]

This is part of doing #395.

To make the raw Inmate data useful and to support need to populate the V2.0 API database, the data must be backed filled, to Auguts 17th, 2013.

The rational for the starting date and one design, not the design that has to be used, on how to do this is located on the Populating-the-Raw-Inmate-Data-from-the-V1.0-API-Database page.

[wilbertom]

I should finish this tonight or tomorrow night.

[bepetersn]

This whole issue?! That would be great!

On Mon, Jun 30, 2014 at 9:42 AM, Wilberto Morales [email protected]
wrote:

I should finish this tonight or tomorrow night.

—
Reply to this email directly or view it on GitHub
#448 (comment).

Brian Peterson

[bepetersn]

Something weird happened on the 27th of June, that resulted in no data dump being made for that date of the scraping run. (See future issue...) However, the scraper didn't totally fail, and the v1 database contains this data.

So I would request that this data be recreated in the same fashion as this issue requires--whatever program is made to fulfill this issue should be kept around, certainly until we know what caused this failure, and how to ensure it doesn't happen again.

[wilbertom]

I made some progress on this but it's really really slow. Tonight I'll reread it to make sure it's right and work on making it faster.

https://github.com/wilbertom/cookcountyjail/tree/backfill

@bepetersn do we have a traceback of the error?

[wilbertom]

Never mind I know how to make it faster. So much for sleeping lol

[wilbertom]

This is harder than I thought. It's taking too much time to dump. I'm starting a new branch.

https://github.com/wilbertom/cookcountyjail/blob/backfill-b/countyapi/management/commands/dump_to_csv.py

[wilbertom]

I'll make a writeup on the google groups to get some feedback.