Cycle over data time

[jfrost-mo]

What problem does your feature request solve?

Currently we cycle over validity time. This has several drawbacks:

• We cannot aggregate over time in the first half of the recipe, which is a common operation. The makes writing recipes harder than it needs to be.
• The individual tasks are very short, which means we actually spend most of the time queuing.
• Looping over non-contiguous case studies is difficult, due to needing to skip time.

Describe the solution you'd like

We should switch to cycling over data time (AKA forecast initiation time). This will allow us to processes multiple case studies in the same CSET run. We should also run larger tasks, where an entire forecast is processed in one go.

Describe alternatives you've considered

[jfrost-mo]

Some guidance on cycling: https://engage.cloud.microsoft/main/org/metoffice.gov.uk/threads/eyJfdHlwZSI6IlRocmVhZCIsImlkIjoiMjg5ODY1MzQ1MDczMTUyMCJ9

[jfrost-mo]

For basic difference plots provide a list of models you want to compare (e.g. [1, 2, 3]) and then do the combination of all of those. Can just arbitrary decide the comparison order based on model number. E.g:

m1 - m2
m1 - m3
m2 - m3

We can then use surface field table to ensure are comparing the same fields.

Table of toggles of models on both axis as way to do it?

[jfrost-mo]

I've now done all of the tasks I identified as part of this work. I'm a bit tempted to try implementing a difference plot to confirm that is now possible, but that should land as a later PR.

So to do on this PR now:

• Split out any separate changes into their own smaller PRs.
• Remove now unneeded recipes, such as LFRic specific ones. (Split into own PR too)
• Convert all existing recipes and include files:
  • cset-workflow/includes/transect.cylc
  • cset-workflow/includes/basic_qq_plot.cylc (Pushed to Convert Q-Q plot to use multi-model cycling more effectively #808)
• Ensure tests cover added code.
• Update documentation to cover new cycling and added code. (It would be nice if the rose edit was documented as part of this.)
• Get some reviews.

[jfrost-mo]

I've broken the pre-processor part of this changeset into issue #807.
And the Q-Q plot conversion into issue #808.