Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cycle over data time #750

Closed
jfrost-mo opened this issue Jul 23, 2024 · 6 comments · Fixed by #765
Closed

Cycle over data time #750

jfrost-mo opened this issue Jul 23, 2024 · 6 comments · Fixed by #765
Assignees
Labels
enhancement New feature or request

Comments

@jfrost-mo
Copy link
Member

What problem does your feature request solve?

Currently we cycle over validity time. This has several drawbacks:

  • We cannot aggregate over time in the first half of the recipe, which is a common operation. The makes writing recipes harder than it needs to be.
  • The individual tasks are very short, which means we actually spend most of the time queuing.
  • Looping over non-contiguous case studies is difficult, due to needing to skip time.

Describe the solution you'd like

We should switch to cycling over data time (AKA forecast initiation time). This will allow us to processes multiple case studies in the same CSET run. We should also run larger tasks, where an entire forecast is processed in one go.

Describe alternatives you've considered

@jfrost-mo jfrost-mo added the enhancement New feature or request label Jul 23, 2024
@jfrost-mo jfrost-mo self-assigned this Jul 23, 2024
@jfrost-mo

This comment was marked as resolved.

@jfrost-mo

This comment was marked as resolved.

@jfrost-mo
Copy link
Member Author

@jfrost-mo
Copy link
Member Author

jfrost-mo commented Aug 2, 2024

For basic difference plots provide a list of models you want to compare (e.g. [1, 2, 3]) and then do the combination of all of those. Can just arbitrary decide the comparison order based on model number. E.g:

m1 - m2
m1 - m3
m2 - m3

We can then use surface field table to ensure are comparing the same fields.

Table of toggles of models on both axis as way to do it?

@jfrost-mo
Copy link
Member Author

jfrost-mo commented Aug 9, 2024

I've now done all of the tasks I identified as part of this work. I'm a bit tempted to try implementing a difference plot to confirm that is now possible, but that should land as a later PR.

So to do on this PR now:

  • Split out any separate changes into their own smaller PRs.
  • Remove now unneeded recipes, such as LFRic specific ones. (Split into own PR too)
  • Convert all existing recipes and include files:
  • Ensure tests cover added code.
  • Update documentation to cover new cycling and added code. (It would be nice if the rose edit was documented as part of this.)
  • Get some reviews.

@jfrost-mo
Copy link
Member Author

jfrost-mo commented Aug 23, 2024

I've broken the pre-processor part of this changeset into issue #807.
And the Q-Q plot conversion into issue #808.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
1 participant