Scraper regression tester?

[estaub]

In working on scrapers I often find that they could use some serious refactoring, but I'm too scared of breaking something to take it on. It seems like some kind of regression testing framework might help.

I'm thinking of a framework that would run a current subscraper, then run the new subscraper, and compare for unexpected differences. By "subscraper" I mean e.g. a bill scraper, vote scraper, people scraper, or whatever.

This would obviously be for manual use in the development process, not e.g. from Travis CI testing.

Does such a thing exist already somewhere?

This may belong better in pupa; I think it makes sense to rough it out here, first, though.

[showerst]

I don't think anything like this exists right now -- there's sort of two different use cases:

1. Refactoring the existing code without adding/removing. -- One way to do this is to have both output JSON files with --scrape and then use a script to compare the outputs. It's slightly hampered by python's json encoding not being idempotent (it often outputs objects with keys in different order from run to run), but I think they may have fixed this in 3.7? I usually do this by hand by side-by-siding them in vscode, but if you ran it through a predictable json serializer one could just md5 hash 'em. Even without you could write a script that parses both and walks the keys. I believe GH has some code to alter the json outputter to always emit keys in the same order, I'll look for that.

2. Adding new bits -- Similar to 1, we just couldn't hash, it would have to be a 'walk the keys and highlight differences' approach.

[estaub]

@showerst Thanks. I've done similar manual things, but of course I'm looking to scale up.

I was thinking of two parsing passes.
A first pass would build up a mapping of pseudo-ids to filenames for the files in each fileset.
The two sets of pseudo-ids would be diffed and any differences reported.

Using the maps from the first pass, a second pass would diff each pair of pseudo-id-matching files and report differences, ignoring timestamps.

Diffing could be done by something like deepdiff.

Does that make sense?

[jamesturk]

hoping to integrate this as part of openstates/issues#85