Regression testing

[KellyStathis]

No description provided.

[axfelix]

This would be relatively simple to implement if we had a non-production server that scheduled harvester runs and logged when results are missing -- we currently do this in production to an extent but it should really be something that blocks commits to master.

[KellyStathis]

In theory, would we limit the testing to only harvest certain repositories, or a limited number of records from each? I could see it taking a while to approve a commit if it does a full run each time. I've never set up anything like this before though!

[axfelix]

I would want to see at least a representative sample, if not a full-full run, every time we try to push changes to master, yeah.

[axfelix]

Best idea so far is to, through the existing Jenkins instance set up for Radiam/FRDR, do a full run against a new SQLite and diff it against the previous master's full-run SQLite, every time something is merged into master. This would be fairly doable I think, would take a bit of work.

[axfelix]

FYI so I don't forget -- this is mostly set up and working on https://jenkins.frdr.ca:9443/login?from=%2Fjob%2Ffrdr_harvest%2F, but the CI hooks aren't firing. Need to look into that.

[axfelix]

We should also implement checking all of our local URLs and ensuring they don't return 404s, which will help test when our deletes aren't persisting to Globus properly, though this should probably be a separate job -- I have the jenkins link above set up to trigger on pull requests, though the delete checking should probably just run on a regular cron.