Jupyter docker: new full build with latest of almost everything except xclim and ravenpy to smooth transition

[tlvu]

Overview

This new full build has latest of almost everything except xclim and ravenpy as intermediate step to smooth transition to pandas 2.2 freq strings changes.

Changes

• New: save conda env export, DockerHub build logs and Jenkins test result in the repo to track changes much more easily between releases

• Jenkins: add SAVE_RESULTING_NOTEBOOK_TIMEOUT for slow notebooks or slow machine

• Jupyter env changes:

  • add conda-pack so we can export the conda env outside of the docker image if need to run locally without docker
  • upgrade from Python 3.9 to 3.11
  • Relevant changes (alphabetical order):

-  - birdy=0.8.4=pyh1a96a4e_0
+      - birdhouse-birdy==0.8.7

# major upgrade from v2 to v3
-  - bokeh=2.4.3=pyhd8ed1ab_3
+  - bokeh=3.4.1=pyhd8ed1ab_0

-  - cartopy=0.21.1=py39h6e7ad6e_0
+  - cartopy=0.23.0=py311h320fe9a_0

-  - cf_xarray=0.8.0=pyhd8ed1ab_0
+  - cf_xarray=0.9.0=pyhd8ed1ab_0

-  - cfgrib=0.9.10.4=pyhd8ed1ab_0
+  - cfgrib=0.9.11.0=pyhd8ed1ab_0

-  - cftime=1.6.2=py39h2ae25f5_1
+  - cftime=1.6.3=py311h1f0f07a_0

-  - climpred=2.3.0=pyhd8ed1ab_0
+  - climpred=2.4.0=pyhd8ed1ab_0

-  - clisops=0.9.6=pyh1a96a4e_0
+  - clisops=0.13.0=pyhca7485f_0

-  - dask=2023.5.1=pyhd8ed1ab_0
+  - dask=2024.5.0=pyhd8ed1ab_0

-  - geopandas=0.13.0=pyhd8ed1ab_0
+  - geopandas=0.14.4=pyhd8ed1ab_0

-  - hvplot=0.8.3=pyhd8ed1ab_0
+  - hvplot=0.9.2=pyhd8ed1ab_0

-  - numpy=1.23.5=py39h3d75532_0
+  - numpy=1.24.4=py311h64a7726_0

-  - numba=0.57.0=py39hb75a051_1
+  - numba=0.59.1=py311h96b013e_0

# major upgrade from v1 to v2
-  - pandas=1.3.5=py39hde0f152_0
+  - pandas=2.1.4=py311h320fe9a_0

# major upgrade to v1
-  - panel=0.14.4=pyhd8ed1ab_0
+  - panel=1.4.2=pyhd8ed1ab_0

# major upgrade from v1 to v2
-  - pydantic=1.10.8=py39hd1e30aa_0
+  - pydantic=2.7.1=pyhd8ed1ab_0

# Python 3.9 to 3.11
-  - python=3.9.16=h2782a2a_0_cpython
+  - python=3.11.6=hab00c5b_0_cpython

-  - raven-hydro=0.2.1=py39h8e2dbb5_1
+  - raven-hydro=0.2.4=py311h64a4d7b_0

-  - ravenpy=0.12.1=py39hf3d152e_0
+      - ravenpy==0.13.1

-  - rioxarray=0.14.1=pyhd8ed1ab_0
+  - rioxarray=0.15.5=pyhd8ed1ab_0

-  - roocs-utils=0.6.4=pyh1a96a4e_0
+  - roocs-utils=0.6.8=pyhd8ed1ab_0

-  - scipy=1.9.1=py39h8ba3f38_0
+  - scipy=1.13.0=py311h517d4fd_1

-  - xarray=2023.1.0=pyhd8ed1ab_0
+  - xarray=2023.8.0=pyhd8ed1ab_0

-  - xclim=0.43.0=py39hf3d152e_1
+  - xclim=0.47.0=py311h38be061_0

-  - xesmf=0.7.1=pyhd8ed1ab_0
+  - xesmf=0.8.5=pyhd8ed1ab_0

-  - xskillscore=0.0.24=pyhd8ed1ab_0
+  - xskillscore=0.0.26=pyhd8ed1ab_0

+  - xscen=0.8.2=pyhd8ed1ab_0

+      - figanos==0.3.0

-      - xncml==0.2
+      - xncml==0.4.0

Test

• Deployed as "beta" image in production for bokeh visualization performance regression testing.
• Manual test notebook https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb for bokeh visualization performance and it looks fine.
• Jenkins build:
  • Default notebooks, all passed: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/54792e6510adfcd1bb21e1bd31fdfa36c5c634e0/docker/saved_buildout/jenkins-buildlogs-default.txt
  • Raven notebooks, only known HydroShare_integration.ipynb failing: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/jenkins-buildlogs-raven.txt

Related Issue / Discussion

• Matching notebook fixes:

  • Pavics-sdi: PR refresh nbs, fix subset-user-input.ipynb pavics-sdi#321
  • Finch: PR url: None
  • PAVICS-landing: PR update nbs, tutorial data and html versions PAVICS-landing#78
  • RavenPy: PR WPS returns datasets : access band_data variable CSHS-CWRA/RavenPy#356
  • Resolves Nb 4 do not work with Python 3.10 (ValueError: popmean.shape[axis] must equal 1.) PAVICS-landing#65
  • Resolves Nb 3 raise wierd warning errors with newer Jupyter env PAVICS-landing#66

• Deployment to PAVICS: Jupyter env: new full build with latest of almost everything bird-house/birdhouse-deploy#453

• Jenkins-config changes for new notebooks: PR url: None

• Other issues found while working on this one

  • --sanitize-with option seems to be behaving weirdly with "newly computed (test) output" computationalmodelling/nbval#204
  • Lost option "Extract Archive" with version 3.4.0 jupyterlab-contrib/jupyter-archive#132
  • Python3.10 builds are failing CSHS-CWRA/RavenPy#357
  • duplicate method does not work as expected CSHS-CWRA/RavenPy#361
  • Resolve HydroShare notebook authentication issue CSHS-CWRA/RavenPy#362

• Previous release: PR Jupyter docker: incremental build to get latest unreleased threddsclient for Thredds 5 compat #134

Additional Information

Full diff conda env export:
81deb99...931cfc9#diff-e8f2a6a53085ae29bb7cedc701c1d345a330651ae971555e85a5c005e94f4cd9

Full new conda env export:
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/conda-env-export.yml

DockerHub build log
https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/docker-buildlogs.txt

[tlvu]

@tlogan2000 The new Jupyter env is live. You can send announcement to our users to restart their server to get the new version.

The previous "stable" version is now the new "previous" version. I've also added a "last-py38-stable-version" to preserve our last Python 3.8 env in case our user need an even older version to reproduce their work.

[Zeitsperre]

We need to cull a few of those entries. beta, alpha, gamma, and last are a lot of development options for images.

[tlvu]

We need to cull a few of those entries. beta, alpha, gamma, and last are a lot of development options for images.

Well beta, alpha, gamma are placeholders when we have many test candidates with slight variations to test for bug fixes. As we have just experienced, it's pretty labor intensive to release a fully tested/certified Jupyter env. So all those placeholders does help during the testing phase.

Previously we deploy those test versions on our staging host but since the staging host do not have all the data, we could not fully test those environments. And the only 100% sure way to not have any surprise is to test directly on the production host ! I've seen case where the JupyterHub is unable to launch the JupyterLab server ! Having the test env on the production host eliminate this issue.

All the app specific envs are for our apps to reproduce the exact env of their runtime. Not all of them, as you can see, are on the "Stable" image. Since their have their own version, they are not force to update at the same time we release a new Stable version. Just ensuring all the notebooks passed is way too labor intensive, I do not want to add all the apps to the certification list !

The last one are an ancien remark from David. We will eventually have last-py39, last-py311 ...

So that's the reason for all those multiple entries.

[tlvu]

We need to cull a few of those entries. beta, alpha, gamma, and last are a lot of development options for images.

Well beta, alpha, gamma are placeholders when we have many test candidates with slight variations to test for bug fixes. As we have just experienced, it's pretty labor intensive to release a fully tested/certified Jupyter env. So all those placeholders does help during the testing phase.

Previously we deploy those test versions on our staging host but since the staging host do not have all the data, we could not fully test those environments. And the only 100% sure way to not have any surprise is to test directly on the production host ! I've seen case where the JupyterHub is unable to launch the JupyterLab server ! Having the test env on the production host eliminate this issue.

All the app specific envs are for our apps to reproduce the exact env of their runtime. Not all of them, as you can see, are on the "Stable" image. Since their have their own version, they are not force to update at the same time we release a new Stable version. Just ensuring all the notebooks passed is way too labor intensive, I do not want to add all the apps to the certification list !

The last one are an ancien remark from David. We will eventually have last-py39, last-py311 ...

So that's the reason for all those multiple entries.

Oh forgot to add that we are trying to release more often so we are continuous trying to test newer versions. Those beta, alpha, gamma will continuously be useful.