Add a minimal test of entire workflow and run it continuously

[timtroendle]

A continuous integration test would catch errors early. For that, the workflow must be 100% automatic and we should have a configuration that requires minimal downloads and minimal runtime. We can then use a simple GitHub action that runs Snakemake with this configuration (example).

[timtroendle]

Here's a list of things that need to be solved (sorted by severity) so that we have a 100% automatic, low data, low runtime workflow test:

• Smaller capacity factor data (limited to geographic or temporal scope).
• Automatic download of EEZ data.
• Smaller or faster load data download (limited to geographic or temporal scope).
• Smaller or faster runoff data download (geographic and temporal scope can be limited today, but download can still be slow when queue is long).
• Handle Gurobi not being available.

The first point (capacityfactors) is by far the worst. When we solve the issues in the list above, all other steps should be very fast, especially when we limit the geographic scope.

[brynpickering]

Agreed that we need this, but limiting scope also creates issues with not picking up some of the more annoying elements of the workflow that we probably want to be testing (e.g. handling Kosovo, filling missing load data, getting ISO vs. EU country codes, assigning plants to basins when they don't fall in one). I can see a different geographic region being needed to catch each of these...

[timtroendle]

You are right. Still, any error that is caught by CI is good. So just because the CI test wouldn't catch all errors doesn't mean it's useless.

We can also apply an hierarchical approach: a simplified workflow that is run very often, and a full workflow that is run less often.

[timtroendle]

We have a similar issue for the estimation of solar and wind potentials. The discussion over there is relevant here, too.
calliope-project/solar-and-wind-potentials#11

[timtroendle]

Bryn mentioned in the dev call today that data caching is possible on GitHub runners.