Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemented fully lazy climate_statistics #1194

Merged
merged 6 commits into from
Jun 30, 2021
Merged

Conversation

schlunma
Copy link
Contributor

@schlunma schlunma commented Jun 25, 2021

While working on #1193 I found that the preprocessor climate_statistics is not lazy if period='full' and a weighted operator is used. The reason is the use of iris.util.broadcast_to_shape here, which broadcasts the time weights to the original shape of the cube.

However, since the release of iris v3, this broadcasting is no longer necessary. PR SciTools/iris#3943 introduced the support of 1D weights for collapsing cubes. This PR implements this for our preprocessor.

Note that while our implementation might have worked previously for multidimensional time arrays, the calculation of the weights didn't make sense at all. Therefore, in this new implementation an error is raised if the time coordinate is not 1D. However, I don't even know if there is any use case for this.

I tested some recipes and all worked fine 👍 I also didn't modify the climate_statistics tests at all, everything worked fine with the new implementation.

Closes #336.

Link to documentation:


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.


To help with the number pull requests:

@schlunma schlunma added the preprocessor Related to the preprocessor label Jun 25, 2021
@schlunma schlunma self-assigned this Jun 25, 2021
@codecov
Copy link

codecov bot commented Jun 25, 2021

Codecov Report

Merging #1194 (09f2809) into main (3b653c4) will decrease coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1194      +/-   ##
==========================================
- Coverage   85.51%   85.51%   -0.01%     
==========================================
  Files         188      188              
  Lines        9147     9146       -1     
==========================================
- Hits         7822     7821       -1     
  Misses       1325     1325              
Impacted Files Coverage Δ
esmvalcore/preprocessor/_time.py 94.13% <100.00%> (-0.03%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3b653c4...09f2809. Read the comment docs.

@schlunma
Copy link
Contributor Author

I think codecov can be ignored here: The coverage of the changed function is 100%, but the relative coverage of the entire file decreased since the total number of lines decreased (and thus the number of hits)...Would be nice if codecov was smart enough to catch this!

Copy link
Member

@bouweandela bouweandela left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I'll have a look at improving the codecov configuration so the checks are useful, see #1195

@zklaus zklaus added this to the v2.4.0 milestone Jun 27, 2021
@schlunma schlunma mentioned this pull request Jun 28, 2021
10 tasks
@schlunma schlunma merged commit 474f71d into main Jun 30, 2021
@schlunma schlunma deleted the lazy_climate_statistics branch June 30, 2021 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preprocessor Related to the preprocessor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Time preprocessor uses numpy array instead of dask array
3 participants