Required update to handle new QA4SM netCDF structure according to Jir…

…a-744 (#77) (#83) * intial * started adapting * adapted existing code to properly handle temporal sub-windows, including renaming the output files and figure titles accordingly * before testing * fig titles and filenames now always contain the default or temporal sub-window case's name. also, maps are only created fo r the default case * intermediate saving * added all code required to generate comparison boxplots * fully integrated compariosn plots and all plots now have the logo instead of the watermark * all statistics .csv files are now stroed in a dedicated zip archive * intermediate commit * Fix bugs with metadata plots * automatic extraction of temporal sub-window names from provided netcdf file and enhanced type hints * testing of tcol metrics plotting * removed debugging statements, implemented bug fix for metadata plot generationwit tcol metrics * comparison boxplots of tcol metrics are not created * implemented saving of created figures in one go for multiple filetypes * code for netcdf file transcription to new qa4sm nc structure * removed accidentally staged restructured files * implemented means for transcription for all test files * added automatic netCDF file transcription prior to tests and corrected for temp. sub-win. dimension as additional item in `QA4SMImage.varnames` in `test_image.py::test_load_vars` * pt2: added automatic netCDF file transcription prior to tests and corrected for temp. sub-win. dimension as additional item in `QA4SMImage.varnames` in `test_image.py::test_load_vars` * netcdf_transcription.py and intra_annual_temp_windows.py now are part of the `qa4sm_reader` package, isntead of `qa4sm`. Further, the globals file was adapted accordingly and now contains all vars required. `qa4sm` imports these specific vars then from the `qa4sm_reader.globals` file. Lastly, `qa4sm_reader.img.QA4SMImg` was adapted to check if a dimension corresponding to temporal sub-windows exists in the specified netCDF file and if not automatically transcribes the loaded dataset. Tests were adapted accordingly. * code cleaning. * code cleaning * introduced function `qa4sm_reader.utils.filter_out_self_combination_tcmetric_vars()` to exclude TC metric vars of same reference and metric dataset from being plotted * added pytesmo dependency * added MANIFEST.in to correctly install the static dir requried for inclusion of logo as watermark * deleted superflous files * adapted packaging of reader to incorporat the static dir containing the logo * tests for intra_annual_temp_windows * added tests to test_utils and made sure they run first * started test_netcdf_transcription.py * fixed test_logo_exists * fixing test_keep_pytesmo_ncfile * tests for netcdf_transcription * finished test development for intra annual metrics and adapation of existing tests * Update environment.yml (#81) * Update environment.yml checking if defining pyscaffold in the env helps fixing the problem with failing tests * Update CI and dependency list * Update env * Update CI * Update env * Update CI --------- * added test to check intra-annual metric related plots and proper file transcription * fixed tests/test_netcdf_transcription.py::test_correct_file_transcription * fixed tests/test_netcdf_transcription.py::test_correct_file_transcription * fixed windows test failing * fixed mamba workflow issue * bug fix for obtaining amount of chars from Path object --------- Co-authored-by: Nicolas F. Bader <[email protected]> Co-authored-by: Monika Tercjak <[email protected]>
awst-austria · Oct 10, 2024 · bc04612 · bc04612
1 parent fa9804f
commit bc04612
Show file tree

Hide file tree

Showing 29 changed files with 4,632 additions and 302 deletions.
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
@@ -14,70 +14,70 @@ on:
         - cron: '0 0 * * *' # daily
 
 jobs:
-    build:
-        name: Build py${{ matrix.python-version }} @ ${{ matrix.os }} 🐍
-        runs-on: ${{ matrix.os }}
-        strategy:
-            matrix:
-                python-version: ['3.8', '3.9', '3.10']
-                os: ['ubuntu-latest']
-                ymlfile: ['environment.yml']
-                include:
-                    - os: 'windows-latest'
-                      python-version: '3.10'
-                      ymlfile: 'environment.yml'
-        steps:
-            - uses: actions/checkout@v2
-              with:
-                  submodules: true
-            - uses: conda-incubator/setup-miniconda@v3
-              with:
-                  miniconda-version: 'latest'
-                  auto-update-conda: true
-                  python-version: ${{ matrix.python-version }}
-                  environment-file: ${{ matrix.ymlfile }}
-                  activate-environment: qa4sm_reader # todo: must match with name in environment.yml
-                  auto-activate-base: false
-            - name: Print environment infos
-              shell: bash -l {0}
-              run: |
-                  conda info -a
-                  conda list
-                  pip list
-                  which pip
-                  which python
-            - name: Export Environment
-              shell: bash -l {0}
-              run: |
-                  mkdir -p artifacts
-                  filename=env_py${{ matrix.python-version }}_${{ matrix.os }}.yml
-                  conda env export --no-builds | grep -v "prefix" > artifacts/$filename
-            - name: Upload Artifacts
-              uses: actions/upload-artifact@v4
-              with:
-                  name: Artifacts-py${{ matrix.python-version }}-${{ matrix.os }}
-                  path: artifacts/*
-            - name: Install package and test
-              shell: bash -l {0}
-              run: |
-                  pip install -e .
-                  pytest
-            - name: Upload Coverage
-              shell: bash -l {0}
-              run: |
-                  pip install coveralls && coveralls --service=github
-              env:
-                  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-                  COVERALLS_FLAG_NAME: ${{ matrix.python-version }}
-                  COVERALLS_PARALLEL: true
-    coveralls:
-        name: Submit Coveralls 👚
-        needs: build
-        runs-on: ubuntu-latest
-        container: python:3-slim
-        steps:
-            - name: Finished
-              run: |
-                  pip3 install --upgrade coveralls && coveralls --service=github --finish
-              env:
-                  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+  build:
+    name: Build py${{ matrix.python-version }} @ ${{ matrix.os }} 🐍
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        python-version: [ '3.8', '3.9', '3.10' ]
+        os: ["ubuntu-latest"]
+        ymlfile: ["environment.yml"]
+        include:
+          - os: "windows-latest"
+            python-version: "3.10"
+            ymlfile: "environment.yml"
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - uses: conda-incubator/setup-miniconda@v3
+        with:
+          miniconda-version: "latest"
+          auto-update-conda: true
+          python-version: ${{ matrix.python-version }}
+          environment-file: ${{ matrix.ymlfile }}
+          activate-environment: qa4sm_reader # todo: must match with name in environment.yml
+          auto-activate-base: false
+      - name: Print environment infos
+        shell: bash -l {0}
+        run: |
+          conda info -a
+          conda list
+          pip list
+          which pip
+          which python
+      - name: Export Environment
+        shell: bash -l {0}
+        run: |
+          mkdir -p artifacts
+          filename=env_py${{ matrix.python-version }}_${{ matrix.os }}.yml
+          conda env export --no-builds | grep -v "prefix" > artifacts/$filename
+      - name: Upload Artifacts
+        uses: actions/upload-artifact@v4
+        with:
+          name: Artifacts-py${{ matrix.python-version }}-${{ matrix.os }}
+          path: artifacts/*
+      - name: Install package and test
+        shell: bash -l {0}
+        run: |
+          pip install -e .
+          pytest
+      - name: Upload Coverage
+        shell: bash -l {0}
+        run: |
+          pip install coveralls && coveralls --service=github
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          COVERALLS_FLAG_NAME: ${{ matrix.python-version }}
+          COVERALLS_PARALLEL: true
+  coveralls:
+    name: Submit Coveralls 👚
+    needs: build
+    runs-on: ubuntu-latest
+    container: python:3-slim
+    steps:
+      - name: Finished
+        run: |
+          pip3 install --upgrade coveralls && coveralls --service=github --finish
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.gitignore b/.gitignore
@@ -58,6 +58,7 @@ MANIFEST
 # tests/test_data
 tests/test_data/old
 tests/test_results
+tests/test_qa4sm_data
 
 # personal testing stuff
 test.py
@@ -72,3 +73,4 @@ tests/test_data/out/*
 .coverage*
 .artifacts/*
 .vscode
+.logs
diff --git a/MANIFEST.in b/MANIFEST.in
@@ -0,0 +1 @@
+graft src/qa4sm_reader/static
diff --git a/environment.yml b/environment.yml
@@ -20,3 +20,4 @@ dependencies:
     - pygeogrids
     - pytest
     - pytest-cov
+    - pytesmo
diff --git a/setup.cfg b/setup.cfg
@@ -36,7 +36,8 @@ install_requires =
     parse
     scipy
     pygeogrids
-
+    pytesmo
+
 # The usage of test_requires is discouraged, see `Dependency Management` docs
 # tests_require = pytest; pytest-cov
 # Require a specific Python version, e.g. Python 2.7 or >= 3.4

diff --git a/src/qa4sm_reader/comparing.py b/src/qa4sm_reader/comparing.py
@@ -21,9 +21,9 @@ class QA4SMComparison:
     take some time, the class can be updated keeping memory of what has already been initialized
     """
     def __init__(self,
-                 paths: list or str,
+                 paths: Union[list, str],
                  extent: tuple = None,
-                 get_intersection: bool = True):
+                 get_intersection: bool = True) -> None:
         """
         Initialise the QA4SMImages from the paths to netCDF files specified
 

diff --git a/src/qa4sm_reader/custom_intra_annual_windows_example.json b/src/qa4sm_reader/custom_intra_annual_windows_example.json
@@ -0,0 +1,92 @@
+{
+    "seasons": {
+        "S1": [
+            [12, 1],
+            [2, 28]
+        ],
+        "S2": [
+            [3, 1],
+            [5, 31]
+        ],
+        "S3": [
+            [6, 1],
+            [8, 31]
+        ],
+        "S4": [
+            [9, 1],
+            [11, 30]
+        ]
+    },
+    "months": {
+        "Jan": [
+            [1, 1],
+            [1, 31]
+        ],
+        "Feb": [
+            [2, 1],
+            [2, 28]
+        ],
+        "Mar": [
+            [3, 1],
+            [3, 31]
+        ],
+        "Apr": [
+            [4, 1],
+            [4, 30]
+        ],
+        "May": [
+            [5, 1],
+            [5, 31]
+        ],
+        "Jun": [
+            [6, 1],
+            [6, 30]
+        ],
+        "Jul": [
+            [7, 1],
+            [7, 31]
+        ],
+        "Aug": [
+            [8, 1],
+            [8, 31]
+        ],
+        "Sep": [
+            [9, 1],
+            [9, 30]
+        ],
+        "Oct": [
+            [10, 1],
+            [10, 31]
+        ],
+        "Nov": [
+            [11, 1],
+            [11, 30]
+        ],
+        "Dec": [
+            [12, 1],
+            [12, 31]
+        ]
+    },
+    "custom": {
+        "star wars month": [
+            [5, 1],
+            [5, 31]
+        ],
+        "halloween season": [
+            [10, 1],
+            [10, 31]
+        ],
+        "advent": [
+            [12, 1],
+            [12, 24]
+        ],
+        "movember": [
+            [11, 1],
+            [11, 30]
+        ],
+        "christmas": [
+            [12, 24],
+            [12, 26]
+        ]
+    }
+}