-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python3.10 builds are failing #357
Comments
Jenkins also fail against the latest Jupyter env is also timing out for me, wonder if it's related. Here is the
|
Potential suspects: - pydantic=2.6.4=pyhd8ed1ab_0
+ pydantic=2.7.0=pyhd8ed1ab_0
- rioxarray=0.15.3=pyhd8ed1ab_0
+ rioxarray=0.15.4=pyhd8ed1ab_0
- xesmf=0.8.4=pyhd8ed1ab_1
+ xesmf=0.8.5=pyhd8ed1ab_0 |
@tlvu Do you notice this only happening for Python 3.10? |
I only noticed the failure simply because the upcoming Jupyter env will be python 3.10, instead of 3.9. I do not run Jenkins on multiples flavors of Python. Note it starts failing only since my most recently build. All previous python 3.10 builds were working fine. |
It does not look like one of those package. I took the bad image But interractively in the JupyterLab env, everything just works. Very weird and not helpful. Don't have any specific error to search. |
It's quite frustrating. I can't figure it out either. I've disabled one particularly flaky test in #358 and tried to prevent read access problems, but the issue remains. |
I am getting something here. Running only the hanging notebook in Jenkins, I got Kernel died while it tries to import the various modules at the beginning of the notebook !!! This is so weird.
|
That's essentially the error that we often see when running some of these tests. Look through the failing builds on GitHub, and you'll see that kernels often die randomly. I can't explain it other than my suspicion that it has to do with illegal memory access from an unsafe data read operation. |
For Jenkins, this is not random at all. With the latest build, Kernel died all the time, 100% reproducible !!! All previous builds Kernel do not die. Notebook code did not change so if it is illegal memory access, then something changed somewhere in the environment, not in the notebook code. |
Moreover, it seems to die during the Also why no illegal memory access if run interractively on JupyterLab?! Everything is so weird ! |
FYI, the Beta env on PAVICS has the broken env py310-240419 and the Gamma has the working env py310-240411. |
Just a note that for the Jupyter env, we moved to python 3.11 and the hanging error is gone ! |
…t xclim and ravenpy to smooth transition (#121) # Overview This new full build has latest of almost everything except `xclim` and `ravenpy` as intermediate step to smooth transition to `pandas` 2.2 freq strings changes. ## Changes - New: save conda env export, DockerHub build logs and Jenkins test result in the repo to track changes much more easily between releases - Jenkins: add `SAVE_RESULTING_NOTEBOOK_TIMEOUT` for slow notebooks or slow machine - Jupyter env changes: - add `conda-pack` so we can export the conda env outside of the docker image if need to run locally without docker - upgrade from Python 3.9 to 3.11 - Relevant changes (alphabetical order): ```diff - - birdy=0.8.4=pyh1a96a4e_0 + - birdhouse-birdy==0.8.7 # major upgrade from v2 to v3 - - bokeh=2.4.3=pyhd8ed1ab_3 + - bokeh=3.4.1=pyhd8ed1ab_0 - - cartopy=0.21.1=py39h6e7ad6e_0 + - cartopy=0.23.0=py311h320fe9a_0 - - cf_xarray=0.8.0=pyhd8ed1ab_0 + - cf_xarray=0.9.0=pyhd8ed1ab_0 - - cfgrib=0.9.10.4=pyhd8ed1ab_0 + - cfgrib=0.9.11.0=pyhd8ed1ab_0 - - cftime=1.6.2=py39h2ae25f5_1 + - cftime=1.6.3=py311h1f0f07a_0 - - climpred=2.3.0=pyhd8ed1ab_0 + - climpred=2.4.0=pyhd8ed1ab_0 - - clisops=0.9.6=pyh1a96a4e_0 + - clisops=0.13.0=pyhca7485f_0 - - dask=2023.5.1=pyhd8ed1ab_0 + - dask=2024.5.0=pyhd8ed1ab_0 - - geopandas=0.13.0=pyhd8ed1ab_0 + - geopandas=0.14.4=pyhd8ed1ab_0 - - hvplot=0.8.3=pyhd8ed1ab_0 + - hvplot=0.9.2=pyhd8ed1ab_0 - - numpy=1.23.5=py39h3d75532_0 + - numpy=1.24.4=py311h64a7726_0 - - numba=0.57.0=py39hb75a051_1 + - numba=0.59.1=py311h96b013e_0 # major upgrade from v1 to v2 - - pandas=1.3.5=py39hde0f152_0 + - pandas=2.1.4=py311h320fe9a_0 # major upgrade to v1 - - panel=0.14.4=pyhd8ed1ab_0 + - panel=1.4.2=pyhd8ed1ab_0 # major upgrade from v1 to v2 - - pydantic=1.10.8=py39hd1e30aa_0 + - pydantic=2.7.1=pyhd8ed1ab_0 # Python 3.9 to 3.11 - - python=3.9.16=h2782a2a_0_cpython + - python=3.11.6=hab00c5b_0_cpython - - raven-hydro=0.2.1=py39h8e2dbb5_1 + - raven-hydro=0.2.4=py311h64a4d7b_0 - - ravenpy=0.12.1=py39hf3d152e_0 + - ravenpy==0.13.1 - - rioxarray=0.14.1=pyhd8ed1ab_0 + - rioxarray=0.15.5=pyhd8ed1ab_0 - - roocs-utils=0.6.4=pyh1a96a4e_0 + - roocs-utils=0.6.8=pyhd8ed1ab_0 - - scipy=1.9.1=py39h8ba3f38_0 + - scipy=1.13.0=py311h517d4fd_1 - - xarray=2023.1.0=pyhd8ed1ab_0 + - xarray=2023.8.0=pyhd8ed1ab_0 - - xclim=0.43.0=py39hf3d152e_1 + - xclim=0.47.0=py311h38be061_0 - - xesmf=0.7.1=pyhd8ed1ab_0 + - xesmf=0.8.5=pyhd8ed1ab_0 - - xskillscore=0.0.24=pyhd8ed1ab_0 + - xskillscore=0.0.26=pyhd8ed1ab_0 + - xscen=0.8.2=pyhd8ed1ab_0 + - figanos==0.3.0 - - xncml==0.2 + - xncml==0.4.0 ``` ## Test - Deployed as "beta" image in production for bokeh visualization performance regression testing. - Manual test notebook https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb for bokeh visualization performance and it looks fine. - Jenkins build: - Default notebooks, all passed: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/54792e6510adfcd1bb21e1bd31fdfa36c5c634e0/docker/saved_buildout/jenkins-buildlogs-default.txt - Raven notebooks, only known `HydroShare_integration.ipynb` failing: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/jenkins-buildlogs-raven.txt ## Related Issue / Discussion - Matching notebook fixes: - Pavics-sdi: PR Ouranosinc/pavics-sdi#321 - Finch: PR url: None - PAVICS-landing: PR Ouranosinc/PAVICS-landing#78 - RavenPy: PR CSHS-CWRA/RavenPy#356 - Resolves Ouranosinc/PAVICS-landing#65 - Resolves Ouranosinc/PAVICS-landing#66 - Deployment to PAVICS: bird-house/birdhouse-deploy#453 - Jenkins-config changes for new notebooks: PR url: None - Other issues found while working on this one - computationalmodelling/nbval#204 - jupyterlab-contrib/jupyter-archive#132 - CSHS-CWRA/RavenPy#357 - CSHS-CWRA/RavenPy#361 - CSHS-CWRA/RavenPy#362 - Previous release: PR #134 ## Additional Information Full diff conda env export: 81deb99...931cfc9#diff-e8f2a6a53085ae29bb7cedc701c1d345a330651ae971555e85a5c005e94f4cd9 Full new conda env export: https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/conda-env-export.yml DockerHub build log https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/931cfc924a147d07b59e88badff9f170e852a03b/docker/saved_buildout/docker-buildlogs.txt
I'm not certain why, but all builds of RavenPy running under Python3.10 seem to be failing for a handful of the same tests, those being:
test_hindcasting.py::TestHindcasting::test_climpred_hindcast_verif
, andtest_calibration.py::test_spotpy_calibration[SACSMA]
This might be due to
climpred
or another library. Will see what can be done.The text was updated successfully, but these errors were encountered: