Profile full monthly generation

[aelkiss]

Generating a full hathifile takes many, many hours. It's probably worth seeing what's so slow and seeing if we can speed it up. Could be database queries, could be json parsing (could try https://github.com/anilmaurya/fast_jsonparser)..

[aelkiss]

It looks like it does a database query for the rights db for every item, but it's only used for the access profile and rights update timestamp. We could consider:

1. doing the queries in batches, ideally via the rights api
2. doing a sort of offline join - basically get the zephir file sorted by id, dump the rights database also sorted by id, and then we can iterate through both - this has often been the most performant way I can come up with to handle these kinds of issues. We could potentially add an 'export all' option to the rights API to support this.

[moseshll]

Based on a stupid test run locally with a 22k-line upd file, fast_jsonparser screams. Just looking at the raw log timestaps, what takes just over a minute with JSON.parse takes slightly under two seconds with FastJsonparser.parse. All the rights and collection stuff no-opped out. So the evidence is very much in favor of pursuing this in addition to finding a better rights strategy.

Alas, the default behavior of fast_jsonparser is to symbolize keys, which caused the rest of the process to bail out. Hence the ludicrous speed. In practice it does seem to be faster than the build-in JSON parser so for ongoing experiments I will leave it in place.