Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lazy implementation of multi_model_statistics and ensemble_statistics preprocessors #968

Merged
merged 81 commits into from
Jun 6, 2023
Merged

Lazy implementation of multi_model_statistics and ensemble_statistics preprocessors #968

merged 81 commits into from
Jun 6, 2023

Commits on Jan 28, 2021

  1. First attempt at replacing multimodel stats with a native iris altern… 

    …ative.
    @Peter9192
    Peter9192 committed Jan 28, 2021
    Configuration menu
    Copy the full SHA
    faaaa38 View commit details
    Browse the repository at this point in the history

  2. temporary test draft to derive new tests from

    @Peter9192
    Peter9192 committed Jan 28, 2021
    Configuration menu
    Copy the full SHA
    4e5a675 View commit details
    Browse the repository at this point in the history

  3. Support percentiles

    @Peter9192
    Peter9192 committed Jan 28, 2021
    Configuration menu
    Copy the full SHA
    d5b75b5 View commit details
    Browse the repository at this point in the history

  4. Docstrings and check for no overlap

    @Peter9192
    Peter9192 committed Jan 28, 2021
    Configuration menu
    Copy the full SHA
    98580d1 View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2021

  1. Resolve span gets its own function

    @Peter9192
    Peter9192 committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    a0336cc View commit details
    Browse the repository at this point in the history

  2. fix merge conflict

    Barbara Vreede authored and Barbara Vreede committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    a5322a2 View commit details
    Browse the repository at this point in the history

  3. first set unit tests for multimodel stats and span

    Barbara Vreede authored and Barbara Vreede committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    8bbd878 View commit details
    Browse the repository at this point in the history

  4. Also accept std as valid statistics

    @Peter9192
    Peter9192 committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    8e24ed9 View commit details
    Browse the repository at this point in the history

  5. Another attempt at supporting std

    @Peter9192
    Peter9192 committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    5e63398 View commit details
    Browse the repository at this point in the history

  6. deal better with kwargs

    @Peter9192
    Peter9192 committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    f93d42f View commit details
    Browse the repository at this point in the history

  7. corrected expected test results

    @bvreede
    bvreede committed Jan 29, 2021
    Configuration menu
    Copy the full SHA
    2bff8ef View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2021

  1. masterplan for overhaul of multimodel tests

    @Peter9192
    Peter9192 committed Feb 1, 2021
    Configuration menu
    Copy the full SHA
    b0f93e6 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2021

  1. git stash dump - wip

    @Peter9192
    Peter9192 committed Feb 18, 2021
    Configuration menu
    Copy the full SHA
    973979d View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2021

  1. Merge branch 'master' into mmstats_lazy

    @stefsmeets
    stefsmeets committed Feb 19, 2021
    Configuration menu
    Copy the full SHA
    40d98ab View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2021

  1. Fix bugs to ensure that tests can run

    @stefsmeets
    stefsmeets committed Feb 22, 2021
    Configuration menu
    Copy the full SHA
    930bec9 View commit details
    Browse the repository at this point in the history

  2. Clean up var mapping to iris.analysis

    @stefsmeets
    stefsmeets committed Feb 22, 2021
    Configuration menu
    Copy the full SHA
    e645812 View commit details
    Browse the repository at this point in the history

  3. Update tests

    @stefsmeets
    stefsmeets committed Feb 22, 2021
    Configuration menu
    Copy the full SHA
    d3b31c9 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2021

  1. Fix test cube metadata so that iris will merge them

    @stefsmeets
    stefsmeets committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    a108445 View commit details
    Browse the repository at this point in the history

  2. Update multimodel tests

    @stefsmeets
    stefsmeets committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    24f2270 View commit details
    Browse the repository at this point in the history

  3. Implement edge cases for multimodel tests

    @stefsmeets
    stefsmeets committed Feb 23, 2021
    Configuration menu
    Copy the full SHA
    ff6ade8 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2021

  1. Refactor multimodel tests

    @stefsmeets
    stefsmeets committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    11232c5 View commit details
    Browse the repository at this point in the history

  2. Add test cases for _resolve_span

    @stefsmeets
    stefsmeets committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    e0577b2 View commit details
    Browse the repository at this point in the history

  3. Generate cube using existing function

    @stefsmeets
    stefsmeets committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    c3a9a0a View commit details
    Browse the repository at this point in the history

  4. Fix Codacy issue

    @stefsmeets
    stefsmeets committed Feb 24, 2021
    Configuration menu
    Copy the full SHA
    2aa566a View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2021

  1. Remove redundant test file

    @stefsmeets
    stefsmeets committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    fb33c19 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into mmstats_lazy

    @stefsmeets
    stefsmeets committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    f45438c View commit details
    Browse the repository at this point in the history

  3. Mark tests failing because of inconsistent plev data with xfail

    @stefsmeets
    stefsmeets committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    8bd9858 View commit details
    Browse the repository at this point in the history

  4. Make cache key more readable

    @stefsmeets
    stefsmeets committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    649b8f3 View commit details
    Browse the repository at this point in the history

  5. Compare coord / metadata attributes directly

    @stefsmeets
    stefsmeets committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    554edea View commit details
    Browse the repository at this point in the history

  6. Use allclose instead of almost_equal for array comparison

    @stefsmeets
    stefsmeets committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    ade5f35 View commit details
    Browse the repository at this point in the history

  7. Group iris.analysis functions

    @stefsmeets
    stefsmeets committed Feb 25, 2021
    Configuration menu
    Copy the full SHA
    04dbaf6 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2021

  1. Add tests to make sure returned cubes are still lazy

    @stefsmeets
    stefsmeets committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    59f2564 View commit details
    Browse the repository at this point in the history

  2. Fix bug with check if cubes are already aligned 

    Using `reduce`, this would always fail
    @stefsmeets
    stefsmeets committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    24bcea0 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into mmstats_lazy

    @stefsmeets
    stefsmeets committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    873ddd0 View commit details
    Browse the repository at this point in the history

  4. Remove aux coord after cubes are merged

    @stefsmeets
    stefsmeets committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    1a6695b View commit details
    Browse the repository at this point in the history

  5. Implement lazy alignment for span='overlap'

    @stefsmeets
    stefsmeets committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    9b3708f View commit details
    Browse the repository at this point in the history

  6. Fix validation data

    @stefsmeets
    stefsmeets committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    b07c704 View commit details
    Browse the repository at this point in the history

  7. Use extrapolate extend time points for span='full'

    @stefsmeets
    stefsmeets committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    a43ecb9 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2021

  1. Implement lazy cube extension scheme for span='full'

    @stefsmeets
    stefsmeets committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    2ffcb9a View commit details
    Browse the repository at this point in the history

  2. Fix pep257: D417

    @stefsmeets
    stefsmeets committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    999d514 View commit details
    Browse the repository at this point in the history

  3. Update documentation

    @stefsmeets
    stefsmeets committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    bdfc125 View commit details
    Browse the repository at this point in the history

  4. Metadata tweaks to make it work with sample data / test recipe

    @stefsmeets
    stefsmeets committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    cea76e4 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2021

  1. Merge branch 'master' into mmstats_lazy

    @stefsmeets
    stefsmeets authored Mar 3, 2021
    Configuration menu
    Copy the full SHA
    5b7f833 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2021

  1. Update docstring and add note

    @stefsmeets
    stefsmeets committed Mar 4, 2021
    Configuration menu
    Copy the full SHA
    c19335f View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2021

  1. Apply suggestions from code review 

    Co-authored-by: Peter Kalverla <[email protected]>
    @stefsmeets @Peter9192
    stefsmeets and Peter9192 authored Mar 5, 2021
    Configuration menu
    Copy the full SHA
    34994cc View commit details
    Browse the repository at this point in the history

  2. Update documentation

    @stefsmeets
    stefsmeets committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    9921b33 View commit details
    Browse the repository at this point in the history

  3. Remove unused test code

    @stefsmeets
    stefsmeets committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    e560bcd View commit details
    Browse the repository at this point in the history

  4. Copy cubes to avoid updating them inplace

    @stefsmeets
    stefsmeets committed Mar 5, 2021
    Configuration menu
    Copy the full SHA
    0bd079b View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2021

  1. Add option to rechunk data before computing statistics

    @Peter9192
    Peter9192 committed Mar 11, 2021
    Configuration menu
    Copy the full SHA
    709d849 View commit details
    Browse the repository at this point in the history

  2. Expand options for statistics

    @stefsmeets
    stefsmeets committed Mar 11, 2021
    Configuration menu
    Copy the full SHA
    f9f5600 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2021

  1. Convert input to lazy arrays for memory efficiency 

    A test was added to ensure that lazy in -> lazy out, real in -> real out
    @stefsmeets
    stefsmeets committed Mar 12, 2021
    Configuration menu
    Copy the full SHA
    cf6bc9c View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2021

  1. Outline of workaround for non-lazy iris funcs

    @Peter9192
    Peter9192 committed Mar 15, 2021
    Configuration menu
    Copy the full SHA
    9f53512 View commit details
    Browse the repository at this point in the history

  2. undo accidental changes to tests

    @Peter9192
    Peter9192 committed Mar 15, 2021
    Configuration menu
    Copy the full SHA
    1f1bfcd View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2021

  1. Fix work-around for iris non-lazy aggregators

    @stefsmeets
    stefsmeets committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    62987b6 View commit details
    Browse the repository at this point in the history

  2. Use slice_over to generate time slices

    @stefsmeets
    stefsmeets committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    7687457 View commit details
    Browse the repository at this point in the history

  3. Remove dask config

    @stefsmeets
    stefsmeets committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    3ff0879 View commit details
    Browse the repository at this point in the history

  4. Remove temporary coordinate

    @stefsmeets
    stefsmeets committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    bb8b3cd View commit details
    Browse the repository at this point in the history

  5. Fix func/dim name and docstring

    @stefsmeets
    stefsmeets committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    ee7af3d View commit details
    Browse the repository at this point in the history

  6. Raise error when a single model is passed to multicube statistics

    @stefsmeets
    stefsmeets committed Mar 16, 2021
    Configuration menu
    Copy the full SHA
    3cad48f View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2021

  1. Merge remote-tracking branch 'origin/master' into mmstats_lazy

    @Peter9192
    Peter9192 committed Mar 17, 2021
    Configuration menu
    Copy the full SHA
    2b5c943 View commit details
    Browse the repository at this point in the history

  2. Realize data beforehand if aggregator is not lazy

    @Peter9192
    Peter9192 committed Mar 17, 2021
    Configuration menu
    Copy the full SHA
    9c8eb67 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2021

  1. Stronger separate lazy from non-lazy path

    @Peter9192
    Peter9192 committed Mar 18, 2021
    Configuration menu
    Copy the full SHA
    d76298e View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2021

  1. Merge branch 'master' into mmstats_lazy

    @stefsmeets
    stefsmeets committed Mar 19, 2021
    Configuration menu
    Copy the full SHA
    c4f0ab2 View commit details
    Browse the repository at this point in the history

  2. Make concat dim a global and fix tests

    @stefsmeets
    stefsmeets committed Mar 19, 2021
    Configuration menu
    Copy the full SHA
    db3aa93 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2021

  1. Simplify one-line if statement as per codacy suggestion

    @Peter9192
    Peter9192 committed Mar 30, 2021
    Configuration menu
    Copy the full SHA
    99159ac View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2021

  1. Merge branch origin/main into this branch and resolve conflicts

    @Peter9192
    Peter9192 committed Jun 9, 2021
    Configuration menu
    Copy the full SHA
    3ac43b5 View commit details
    Browse the repository at this point in the history

  2. Revert "Merge branch origin/main into this branch and resolve conflicts" 

    This reverts commit 3ac43b5.
    @Peter9192
    Peter9192 committed Jun 9, 2021
    Configuration menu
    Copy the full SHA
    73840dc View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'origin/main' into mmstats_lazy - second… 

    … attempt
    @Peter9192
    Peter9192 committed Jun 9, 2021
    Configuration menu
    Copy the full SHA
    f47aceb View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2021

  1. Various improvements, move rechunking to before regridding

    @bouweandela
    bouweandela committed Jun 11, 2021
    Configuration menu
    Copy the full SHA
    9d15cea View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2021

  1. Fix typo

    @bouweandela
    bouweandela authored Jun 12, 2021
    Configuration menu
    Copy the full SHA
    e839dbe View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2021

  1. Add unit tests for _rechunk

    @bouweandela
    bouweandela committed Jun 13, 2021
    Configuration menu
    Copy the full SHA
    efbcb34 View commit details
    Browse the repository at this point in the history

  2. Log whether or not data is lazy

    @bouweandela
    bouweandela committed Jun 13, 2021
    Configuration menu
    Copy the full SHA
    78d73b3 View commit details
    Browse the repository at this point in the history

  3. Fix sample data tests

    @bouweandela
    bouweandela committed Jun 13, 2021
    Configuration menu
    Copy the full SHA
    6f6e61f View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2021

  1. Merge branch 'main' of github.com:esmvalgroup/esmvalcore into mmstats… 

    …_lazy
    @bouweandela
    bouweandela committed Jun 14, 2021
    Configuration menu
    Copy the full SHA
    721e255 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2021

  1. Merge branch 'main' of github.com:esmvalgroup/esmvalcore into mmstats… 

    …_lazy
    @bouweandela
    bouweandela committed Jun 22, 2021
    Configuration menu
    Copy the full SHA
    aab318e View commit details
    Browse the repository at this point in the history

  2. Fix some types

    @bouweandela
    bouweandela committed Jun 22, 2021
    Configuration menu
    Copy the full SHA
    0658219 View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2022

  1. Merge branch 'main' of github.com:esmvalgroup/esmvalcore into mmstats… 

    …_lazy
    @bouweandela
    bouweandela committed Jun 28, 2022
    Configuration menu
    Copy the full SHA
    17dd7fd View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2022

  1. Merge branch 'main' of github.com:ESMValGroup/ESMValCore into mmstats… 

    …_lazy
    @bouweandela
    bouweandela committed Sep 23, 2022
    Configuration menu
    Copy the full SHA
    3247d9c View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2023

  1. Merge branch 'main' of github.com:esmvalgroup/esmvalcore into mmstats… 

    …_lazy
    @bouweandela
    bouweandela committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    5b4fecd View commit details
    Browse the repository at this point in the history

  2. Undo needless changes and fix test

    @bouweandela
    bouweandela committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    9039242 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into mmstats_lazy

    @bouweandela
    bouweandela authored Jun 1, 2023
    Configuration menu
    Copy the full SHA
    a364c84 View commit details
    Browse the repository at this point in the history