Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lazy implementation of multi_model_statistics and ensemble_statistics preprocessors #968

Merged
merged 81 commits into from
Jun 6, 2023
Merged

Lazy implementation of multi_model_statistics and ensemble_statistics preprocessors #968

Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
faaaa38
First attempt at replacing multimodel stats with a native iris altern…
Peter9192 Jan 28, 2021
4e5a675
temporary test draft to derive new tests from
Peter9192 Jan 28, 2021
d5b75b5
Support percentiles
Peter9192 Jan 28, 2021
98580d1
Docstrings and check for no overlap
Peter9192 Jan 28, 2021
a0336cc
Resolve span gets its own function
Peter9192 Jan 29, 2021
a5322a2
fix merge conflict
Jan 29, 2021
8bbd878
first set unit tests for multimodel stats and span
Jan 29, 2021
8e24ed9
Also accept std as valid statistics
Peter9192 Jan 29, 2021
5e63398
Another attempt at supporting std
Peter9192 Jan 29, 2021
f93d42f
deal better with kwargs
Peter9192 Jan 29, 2021
2bff8ef
corrected expected test results
bvreede Jan 29, 2021
b0f93e6
masterplan for overhaul of multimodel tests
Peter9192 Feb 1, 2021
973979d
git stash dump - wip
Peter9192 Feb 18, 2021
40d98ab
Merge branch 'master' into mmstats_lazy
stefsmeets Feb 19, 2021
930bec9
Fix bugs to ensure that tests can run
stefsmeets Feb 22, 2021
e645812
Clean up var mapping to `iris.analysis`
stefsmeets Feb 22, 2021
d3b31c9
Update tests
stefsmeets Feb 22, 2021
a108445
Fix test cube metadata so that iris will merge them
stefsmeets Feb 23, 2021
24f2270
Update multimodel tests
stefsmeets Feb 23, 2021
ff6ade8
Implement edge cases for multimodel tests
stefsmeets Feb 23, 2021
11232c5
Refactor multimodel tests
stefsmeets Feb 24, 2021
e0577b2
Add test cases for _resolve_span
stefsmeets Feb 24, 2021
c3a9a0a
Generate cube using existing function
stefsmeets Feb 24, 2021
2aa566a
Fix Codacy issue
stefsmeets Feb 24, 2021
fb33c19
Remove redundant test file
stefsmeets Feb 25, 2021
f45438c
Merge branch 'master' into mmstats_lazy
stefsmeets Feb 25, 2021
8bd9858
Mark tests failing because of inconsistent plev data with xfail
stefsmeets Feb 25, 2021
649b8f3
Make cache key more readable
stefsmeets Feb 25, 2021
554edea
Compare coord / metadata attributes directly
stefsmeets Feb 25, 2021
ade5f35
Use `allclose` instead of `almost_equal` for array comparison
stefsmeets Feb 25, 2021
04dbaf6
Group iris.analysis functions
stefsmeets Feb 25, 2021
59f2564
Add tests to make sure returned cubes are still lazy
stefsmeets Mar 1, 2021
24bcea0
Fix bug with check if cubes are already aligned
stefsmeets Mar 1, 2021
873ddd0
Merge branch 'master' into mmstats_lazy
stefsmeets Mar 1, 2021
1a6695b
Remove aux coord after cubes are merged
stefsmeets Mar 1, 2021
9b3708f
Implement lazy alignment for span='overlap'
stefsmeets Mar 1, 2021
b07c704
Fix validation data
stefsmeets Mar 1, 2021
a43ecb9
Use extrapolate extend time points for span='full'
stefsmeets Mar 1, 2021
2ffcb9a
Implement lazy cube extension scheme for span='full'
stefsmeets Mar 2, 2021
999d514
Fix pep257: D417
stefsmeets Mar 2, 2021
bdfc125
Update documentation
stefsmeets Mar 2, 2021
cea76e4
Metadata tweaks to make it work with sample data / test recipe
stefsmeets Mar 2, 2021
5b7f833
Merge branch 'master' into mmstats_lazy
stefsmeets Mar 3, 2021
c19335f
Update docstring and add note
stefsmeets Mar 4, 2021
34994cc
Apply suggestions from code review
stefsmeets Mar 5, 2021
9921b33
Update documentation
stefsmeets Mar 5, 2021
e560bcd
Remove unused test code
stefsmeets Mar 5, 2021
0bd079b
Copy cubes to avoid updating them inplace
stefsmeets Mar 5, 2021
709d849
Add option to rechunk data before computing statistics
Peter9192 Mar 11, 2021
f9f5600
Expand options for statistics
stefsmeets Mar 11, 2021
cf6bc9c
Convert input to lazy arrays for memory efficiency
stefsmeets Mar 12, 2021
9f53512
Outline of workaround for non-lazy iris funcs
Peter9192 Mar 15, 2021
1f1bfcd
undo accidental changes to tests
Peter9192 Mar 15, 2021
62987b6
Fix work-around for iris non-lazy aggregators
stefsmeets Mar 16, 2021
7687457
Use `slice_over` to generate time slices
stefsmeets Mar 16, 2021
3ff0879
Remove dask config
stefsmeets Mar 16, 2021
bb8b3cd
Remove temporary coordinate
stefsmeets Mar 16, 2021
ee7af3d
Fix func/dim name and docstring
stefsmeets Mar 16, 2021
3cad48f
Raise error when a single model is passed to multicube statistics
stefsmeets Mar 16, 2021
2b5c943
Merge remote-tracking branch 'origin/master' into mmstats_lazy
Peter9192 Mar 17, 2021
9c8eb67
Realize data beforehand if aggregator is not lazy
Peter9192 Mar 17, 2021
d76298e
Stronger separate lazy from non-lazy path
Peter9192 Mar 18, 2021
c4f0ab2
Merge branch 'master' into mmstats_lazy
stefsmeets Mar 19, 2021
db3aa93
Make concat dim a global and fix tests
stefsmeets Mar 19, 2021
99159ac
Simplify one-line if statement as per codacy suggestion
Peter9192 Mar 30, 2021
3ac43b5
Merge branch origin/main into this branch and resolve conflicts
Peter9192 Jun 9, 2021
73840dc
Revert "Merge branch origin/main into this branch and resolve conflicts"
Peter9192 Jun 9, 2021
f47aceb
Merge remote-tracking branch 'origin/main' into mmstats_lazy - second…
Peter9192 Jun 9, 2021
9d15cea
Various improvements, move rechunking to before regridding
bouweandela Jun 11, 2021
e839dbe
Fix typo
bouweandela Jun 12, 2021
efbcb34
Add unit tests for _rechunk
bouweandela Jun 13, 2021
78d73b3
Log whether or not data is lazy
bouweandela Jun 13, 2021
6f6e61f
Fix sample data tests
bouweandela Jun 13, 2021
721e255
Merge branch 'main' of github.com:esmvalgroup/esmvalcore into mmstats…
bouweandela Jun 14, 2021
aab318e
Merge branch 'main' of github.com:esmvalgroup/esmvalcore into mmstats…
bouweandela Jun 22, 2021
0658219
Fix some types
bouweandela Jun 22, 2021
17dd7fd
Merge branch 'main' of github.com:esmvalgroup/esmvalcore into mmstats…
bouweandela Jun 28, 2022
3247d9c
Merge branch 'main' of github.com:ESMValGroup/ESMValCore into mmstats…
bouweandela Sep 23, 2022
5b4fecd
Merge branch 'main' of github.com:esmvalgroup/esmvalcore into mmstats…
bouweandela Jun 1, 2023
9039242
Undo needless changes and fix test
bouweandela Jun 1, 2023
a364c84
Merge branch 'main' into mmstats_lazy
bouweandela Jun 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading