Validation server idea

[stschiff]

The discussion around #83 made me come up with an idea. It would actually be fairly easy to build an HTTP server (deployed on the same cloud-server where the other server currently lives), which does the validation of new branches, PRs and commits for us.

The server would be a simple HTTP server program with a REST API. I can think of the following APIs:

/validate/<branch_name> -> This would trigger the server to pull the branch and perform a validation. Bandwidth usage would be minimal, because it would only have to pull big files if they changed from master or the latest pulls

/get_validation_results/<branch_name> -> This would either respond with some kind of "pending" if it's still running, or "done" with the validation results.

We then use a GitHub Webhook to trigger /validate/<branch_name> whenever a new PR gets opened, or an existing PR gets new commits. A GitHub-action would then query /get_validation_results/<branch_name> for as long as the result is pending, and show the result when its done.

I'm arguing a but from an armchair here, knowing that the HTTP server itself needs maintenance and code updates, and I don't know when I'll find the time to do it, but conceptually it's easy, and it would solve our bandwidth problems and finally also check genotype data.

[stschiff]

Note that this server would live on the same machine as our Poseidon file server, but of course operate in its own directory and under its own port.

I also think that there are two things that we should consider when doing this:

1. containerise the servers in a singularity or docker container to simplify deployment.
2. Factor out the server programs into a separate package poseidon-servers or so.

[nevrome]

Yes - I guess that would work. An alternative, low-tech solution would be to make a PR-checklist (which we need anyway!) for the given reviewer, that then includes pulling the branch and running validate locally. Your solution is of course better, because it removes the human component. But it's also an expensive one regarding time and effort.

[stschiff]

Yes... well, as I said the http server needs updates regarding many of the latest changes, including logging... and I think we also need to tackle the deployment problem, which is currently very manual and non-reproducible. When I'm fixing that, perhaps I get a bit of a taste to continue the web-programming bit of this project.