[Numpy2] Support for numpy==2.0.0
#2395
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2395 +/- ##
=======================================
Coverage 94.77% 94.77%
=======================================
Files 251 251
Lines 14266 14270 +4
=======================================
+ Hits 13520 13524 +4
Misses 746 746 ☔ View full report in Codecov by Sentry. |
|
UGH! This doesn't look good at all https://github.com/ESMValGroup/ESMValCore/actions/runs/8664961338/job/23762564320 |
|
xref Unidata/cftime#318 |
|
Spot on @bouweandela 🍻 Fortunately there is a mature PR for it Unidata/cftime#319 |
|
K then, just waiting for a new |
valeriupredoi
changed the title
numpy==2.0.0rc1numpy==2.0.0rc2
|
still royally hosed even with rc2 |
| # previous to numpy 2.0, a np.array(esmpy_lat.shape) was used | ||
| # numpy>=2.0 throws ValueError: matrix transpose with ndim<2 is undefined | ||
| grid = esmpy.Grid(np.vstack(esmpy_lat.shape), |
There was a problem hiding this comment.
Is this backwards-compatible? If yes, do we need this comment?
There was a problem hiding this comment.
yes, it is, functionally and it returns the same sort of structure, only typed differently
| # explicitly casting to int32 since without it, in Numpy 2.0 | ||
| # one gets OverflowError: Python integer 360 out of bounds for int8 | ||
| # see notes https://numpy.org/devdocs/numpy_2_0_migration_guide.html | ||
| src_points_with_convex_hull[-2 * n_hull:-n_hull, 1] -= np.int32(360) | ||
| src_points_with_convex_hull[-n_hull:, 1] += np.int32(360) |
There was a problem hiding this comment.
Would this work with 360.0 instead of np.int32(360)? The points should be floats, not integers, so to me this makes much more sense. If this works, you can also remove the comment
There was a problem hiding this comment.
not sure, will test - but I agree about points' types indeed
There was a problem hiding this comment.
nopers, a float there throws this:
> src_points_with_convex_hull[-2 * n_hull:-n_hull, 1] -= 360.
E numpy._core._exceptions._UFuncOutputCastingError: Cannot cast ufunc 'subtract' output from dtype('float64') to dtype('int8') with casting rule 'same_kind'
so we either np.int32 the 360 or we np.float the left member
| @@ -920,9 +920,12 @@ def test_season_not_available(self): | |||
| name='clim_season', | |||
| seasons=['JFMAMJ', 'JASOND'], | |||
| ) | |||
|
|
|||
| # numpy>=2.0 these need to be explicitly cast to numpy strings | |||
| two_seasons = [np.str_('JASOND'), np.str_('JFMAMJ')] | |||
There was a problem hiding this comment.
I guess this is not backwards-compatible? How does the error look like? It might make sense to adapt this in the actual code so the error message still looks nice
There was a problem hiding this comment.
yes it is, as it is now, ie the test does what it says on the lid for both numpy<2 and numpy>=2, here's the failed test for numpy>=2 if we don't change to forced typing:
> with pytest.raises(ValueError, match=re.escape(msg)):
E AssertionError: Regex pattern did not match.
E Regex: "Seasons\\ \\('DJF',\\ 'MAM',\\ 'JJA',\\ 'SON'\\)\\ do\\ not\\ match\\ prior\\ season\\ extraction\\ \\['JASOND',\\ 'JFMAMJ'\\]\\."
E Input: "Seasons ('DJF', 'MAM', 'JJA', 'SON') do not match prior season extraction [np.str_('JASOND'), np.str_('JFMAMJ')]."
tests/unit/preprocessor/_time/test_time.py:930: AssertionError
|
Maybe we should wait for Iris to be numpy 2 compatible: SciTools/iris#6035 |
|
It looks like that will happen when Iris 3.11 is released at the earliest, planned in October: SciTools/iris#5571 (comment). |
|
many thanks for the reviews and comments @schlunma and @bouweandela 🍺 x2 I don't think we can wait until October TBF, I say we get this in and see what's the worst it can happen, and fairly sure not much, and if there are problems, we can alert iris and tell them to hurry up 😁 |
I'm sorry, but I have to disagree. What would most likely happen is that users will get wrong results and/or crashes. |
Co-authored-by: Manuel Schlund <[email protected]>
that is indeed the possibility; I guess there's one way to find out about the magnitude of that - run the iris tests with a numpy>=2 pin in an iris fork |
|
Monitoring SciTools/iris#6043 |
|
OK folks, iris have implemented numpy2 support and closed SciTools/iris#6043 - their 3.11 is fast approaching, see https://github.com/SciTools/iris/milestone/51 - so could I please ask you to get back to reviewing this so we can plop it in as soon as the new iris is out? 🍺 The two review chevaliers: @schlunma @bouweandela 🎠 |
Description
This PR introduces the changes we need to support Numpy 2.0 (these changes were collected from running the tests, so they may not be 100% exhaustive, but given that we have 95% test coverage, I'd say they are 95% exhaustive; we can deal with missed cases as they come along, if any; I would not use the ruff plugin test Numpy recommends to migrate, since that will introduce a lot of style changes that we may want to make when we include ruff as our linter). Most changes are in our testing suite, but there are a couple in the source code; these are backwards compatible changes, ie all tests pass when running with numpy=1.26.4
The only test that fails (both for Numpy2 and numpy legacy) is related to the pins on pandas, that need to go since we need to use the latest pandas for environment compatibility, see #2459 ; I'll try fix that issue here, since it's related to (but not a direct consequence of) Numpy2.
Closes #2460
Closes #2459
iris issue to monitor SciTools/iris#6043
-> which that is already in iris==3.11, which leads to
TODO pin
iris>=3.11The second issue closure is more of a brush it under the carpet via a pytest xfail, the original issue that kept creeping up is #2305 and the actual issue is with Pandas, and listed in pandas-dev/pandas#57002 - we can wait on its fix (and have us test quietness via the test
xfail) or @schlunma can propose a more elegant solution in a separate PR?Historical guff
OK @ESMValGroup/technical-lead-development-team we finally have a correct set of tests with the latest Numpy=2.0.0 (stable) - don't mind the ones related to
pandas, I had to unpin it to gte pandas=2.2.2 built with Numpy 2.0 (otherwise the environment wouldn't solve) -> we have some work to do 👍 https://github.com/ESMValGroup/ESMValCore/actions/runs/9582595825/job/26421917358?pr=2395Numpy 2.0 is impending: it is now fully out (see https://anaconda.org/conda-forge/numpy ); resources https://anaconda.org/conda-forge/numpy/files and https://github.com/numpy/numpy/releases - they recommend a test with the RC and list the changes roadmap in https://numpy.org/devdocs/numpy_2_0_migration_guide.html so we can test ourselves. Note: this branch should not be merged! If everything works out fine, we'll just pick up numpy 2.0 as per normal. ATTN: @ESMValGroup/technical-lead-development-teamnumpy=2.0.0rc1on conda-forge? numpy/numpy#26271 it is reported that the channel isconda-forge/label/numpy_devpython-stratify,shapely, andpandas, since most prob they are pinning the upper bound of numpy to <2.0, see GA testscftime, as @bouweandela points out in comments, see Numpy 2.0 incompatibility Unidata/cftime#318 (and linked PR), others are due to actual package compilation against a Numpy=1.x that barfs when it sees 2.x (those should be solved at recompile/redeploy time), and others are due to netCDF4 complaining about dtypes - that needs a closer look