Build with Numpy 2.0 #231

jaimergp · 2024-06-18T16:04:34Z

Investigating #210

jaimergp · 2024-06-18T16:41:14Z

CI available at https://github.com/jaimergp/zfp/actions/runs/9568406928/job/26378542726. It passes but I do see some warnings about deprecated APIs:

[  8%] Building C object python/CMakeFiles/zfpy.dir/zfpy.c.o
In file included from /opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1909,
                 from /opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/numpy/_core/include/numpy/ndarrayobject.h:12,
                 from /opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/numpy/_core/include/numpy/arrayobject.h:5,
                 from /home/runner/work/zfp/zfp/build/python/zfpy.c:1232:
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/numpy/_core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~

[ 98%] Building C object tests/python/CMakeFiles/test_utils.dir/test_utils.c.o
In file included from /opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1909,
                 from /opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/numpy/_core/include/numpy/ndarrayobject.h:12,
                 from /opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/numpy/_core/include/numpy/arrayobject.h:5,
                 from /home/runner/work/zfp/zfp/build/tests/python/test_utils.c:1232:
/opt/hostedtoolcache/Python/3.12.3/x64/lib/python3.12/site-packages/numpy/_core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~

rgommers · 2024-06-18T18:17:39Z

Those warnings are harmless, you can safely ignore them. Would be nice to fix them eventually of course, but they've been there for many years and are unrelated to numpy 2.0

seberg · 2024-06-18T18:58:57Z

Seems these builds are still using Python 3.8, in that case you can't force NumPy>=2 (not available). OTOH, if there is nothing else limiting the version, you'll get Numpy 2 anyway during the build.

Now, in principle if you still build for 3.8 you should still use oldest-supported-numpy unfortunately (although, not usre if you need to rebuild at all for 3.8 at this point).

hmaarrfk · 2024-06-20T12:41:30Z

python/requirements.txt

@@ -1,2 +1,2 @@
 cython>=0.22
-numpy>=1.8.0
+numpy>=2.0


is this a runtime requirements file or a build time requirements file?

jakirkham · 2024-06-24T18:42:10Z

appveyor.yml


    - COMPILER: msvc
      GENERATOR: Visual Studio 14 2015
      PLATFORM: Win32
      BUILD_TYPE: Release
-      PYTHON_VERSION: 38
+      PYTHON_VERSION: 39

    - COMPILER: mingw
      GENERATOR: MinGW Makefiles


Think these MinGW builds also need Python version set to 3.9. Appears they use Python 2.7 otherwise

jakirkham · 2024-06-24T18:42:47Z

Think we also need someone to approve GHA so that it can run

@lindstro would you be able to help with that? 🙂

jaimergp · 2024-06-24T20:26:17Z

The workflow triggers are not configured for pull requests. So you have to go to my branch and see the Actions tab there (push event triggers are active).

jakirkham · 2024-06-24T20:33:14Z

I trust that you got it to work 🙂

Am now trying to figure out how we can work with the maintainers here to integrate it (though the suggested AppVeyor change above still seems advisable)

lindstro · 2024-06-24T20:40:08Z

Think we also need someone to approve GHA so that it can run

@lindstro would you be able to help with that? 🙂

@jakirkham I'm happy to help out, but I must admit to being a Python novice at best. It seems reasonable to bump Python from 3.8 to 3.9, but are there no concerns about requiring NumPy 2.0?

There's also this lingering issue with setup.py that appears to not have gotten fixed. I'm not sure what the intent is. @GarrettDMorrison @da-wad Could either of you comment?

jakirkham · 2024-06-28T05:40:54Z

Right now we are just focused on getting CI running. Think there is a button a repo owner can click

lindstro · 2024-06-28T23:54:20Z

@jakirkham Thanks for the pointer. However, the "approve and run" button in question does not appear visible to me. I'm no t sure if that's some kind of repo configuration issue.

I also don't know why we're not running workflows on pull requests. Let me look into this. I think for now you'll have to consult @jaimergp's fork (see link to CI above).

jaimergp · 2024-07-01T08:21:57Z

I can add the PR support here, I think. Then you'll see the button John mentions.

lindstro · 2024-07-01T14:55:06Z

I can add the PR support here, I think. Then you'll see the button John mentions.

@jaimergp I should have mentioned that I already added workflows on PRs on develop. We actually want this for our feature branches as well, not just develop. But I didn't do anything about concurrency. Can you please explain what this does?

I do see the "Approve and run" button now and tests are running...

william-silversmith · 2024-07-04T02:40:42Z

Just looking at AppVeyor, it kinda looks like the failing runs are using python2.7 (EOL Jan. 1, 2020) and an old pip and so are not finding numpy 2.0 wheels. If its reconfigured to use Python3 and a newer pip is used, it'll probably work.

jaimergp · 2024-07-06T18:15:55Z

We actually want this for our feature branches as well, not just develop.

I was trying to have the workflows triggered for every push to develop (usually a merge), and then every PR targetting develop (from either a branch or a fork). If you don't restrict the push branches, then on PRs you'd duplicate every run because they'd be triggered by both push and pull_request.

But I didn't do anything about concurrency. Can you please explain what this does?

It'll cancel ongoing runs for a given PR (a new push to the PR will cancel the previous runs triggered by previous pushes if they are still running). On develop, it makes sure that only workflow is running per commit (just a safeguard, this is usually the case) anyway.

jaimergp · 2024-07-06T18:16:42Z

Also I'd like to see if you are interested in using Appveyor, given that we can simply configure Github Actions to run on Windows too. Thoughts?

lindstro · 2024-07-06T22:17:59Z

We actually want this for our feature branches as well, not just develop.

I was trying to have the workflows triggered for every push to develop (usually a merge), and then every PR targetting develop (from either a branch or a fork). If you don't restrict the push branches, then on PRs you'd duplicate every run because they'd be triggered by both push and pull_request.

I'm not sure I understand. Clearly workflows are not triggered currently on PRs, which adding pull_request is supposed to address (i.e., what I just did on develop).

But I didn't do anything about concurrency. Can you please explain what this does?

It'll cancel ongoing runs for a given PR (a new push to the PR will cancel the previous runs triggered by previous pushes if they are still running). On develop, it makes sure that only workflow is running per commit (just a safeguard, this is usually the case) anyway.

Gotcha.

lindstro · 2024-07-06T22:25:54Z

Also I'd like to see if you are interested in using Appveyor, given that we can simply configure Github Actions to run on Windows too. Thoughts?

I'd be the first to drop AppVeyor in favor of something better. I just don't have the cycles at the moment to investigate how to transition to GitHub Actions (which compilers and versions of CMake and Python to use, how/whether to deal with Fortran, how to interface with CDash, ...). Keep in mind that we're currently testing both MSVC and MinGW through AppVeyor.

If someone else has time to help out with this, I'd be very grateful.

jaimergp · 2024-07-07T08:57:10Z

I'm not sure I understand. Clearly workflows are not triggered currently on PRs, which adding pull_request is supposed to address (i.e., what I just did on develop).

I reverted the branch filter so you can see what happens when you run the workflows. You will see duplicated (redundant) entries for push and pull_request, which is usually a waste of CI resources.

jaimergp · 2024-07-08T11:42:47Z

I don't know why it can't find the Python.h headers. They are there in include. Those CMake functions are now deprecated so maybe it would be a good idea to upgrade them, but I don't have the time nor the expertise to dive there, sorry.

hmaarrfk · 2024-07-08T12:09:59Z

See #227 or I believe I had a patch:
https://github.com/conda-forge/zfpy-feedstock/blob/main/recipe/fix_find_python_stuff.patch

Use Numpy 2.0

b251f64

jaimergp mentioned this pull request Jun 18, 2024

Numpy 2.0 #210

Open

update this requirement too

859ca72

jaimergp mentioned this pull request Jun 18, 2024

Rebuild for numpy 2.0 conda-forge/zfpy-feedstock#37

Merged

Ask for py39

2fb939d

normanrz mentioned this pull request Jun 19, 2024

Add numpy 2.0 compatibility zarr-developers/numcodecs#535

Merged

hmaarrfk reviewed Jun 20, 2024

View reviewed changes

jakirkham reviewed Jun 24, 2024

View reviewed changes

jakirkham mentioned this pull request Jun 24, 2024

NumPy 2.0 support zarr-developers/numcodecs#521

Closed

jaimergp added 2 commits July 1, 2024 10:24

Add PR triggers for workflows, and control for concurrency

d3ba030

Merge branch 'develop' into numpy2

82e225e

This was referenced Jul 1, 2024

Syntax error in setup.py #233

Closed

fix: syntax error in setup.py + modernize #236

Closed

All pushes trigger workflows

a3e75dc

jaimergp added 16 commits July 7, 2024 11:22

Force Python3 in Appveyor?

14f6c99

enable interactive debugging temporarily

5d92985

Adjust VM images

2af8987

exit if PYTHON_DIR does not exist

b03dc38

python3 doesn't exist; use pip3

b2daf40

negate equality

e284c5b

revert to powershell here

5974a8c

exists -> exist

0335986

adjust generator

65fecfe

use -A instead of architecture in -G

cf073f1

debug again

cd5d726

remove PYTHON_INCLUDE_DIR workarounds

ce547a5

do not debug

f05d9d2

define PYTHON_INCLUDE_DIR

62ba300

pass PYTHON_INCLUDE_DIR

642846c

pass directly

b5212a7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build with Numpy 2.0 #231

Build with Numpy 2.0 #231

jaimergp commented Jun 18, 2024

jaimergp commented Jun 18, 2024

rgommers commented Jun 18, 2024

seberg commented Jun 18, 2024

hmaarrfk Jun 20, 2024

jakirkham Jun 24, 2024

jakirkham commented Jun 24, 2024

jaimergp commented Jun 24, 2024

jakirkham commented Jun 24, 2024

lindstro commented Jun 24, 2024

jakirkham commented Jun 28, 2024

lindstro commented Jun 28, 2024

jaimergp commented Jul 1, 2024

lindstro commented Jul 1, 2024

william-silversmith commented Jul 4, 2024

jaimergp commented Jul 6, 2024

jaimergp commented Jul 6, 2024

lindstro commented Jul 6, 2024

lindstro commented Jul 6, 2024

jaimergp commented Jul 7, 2024

jaimergp commented Jul 8, 2024

hmaarrfk commented Jul 8, 2024

Build with Numpy 2.0 #231

Are you sure you want to change the base?

Build with Numpy 2.0 #231

Conversation

jaimergp commented Jun 18, 2024

jaimergp commented Jun 18, 2024

rgommers commented Jun 18, 2024

seberg commented Jun 18, 2024

hmaarrfk Jun 20, 2024

Choose a reason for hiding this comment

jakirkham Jun 24, 2024

Choose a reason for hiding this comment

jakirkham commented Jun 24, 2024

jaimergp commented Jun 24, 2024

jakirkham commented Jun 24, 2024

lindstro commented Jun 24, 2024

jakirkham commented Jun 28, 2024

lindstro commented Jun 28, 2024

jaimergp commented Jul 1, 2024

lindstro commented Jul 1, 2024

william-silversmith commented Jul 4, 2024

jaimergp commented Jul 6, 2024

jaimergp commented Jul 6, 2024

lindstro commented Jul 6, 2024

lindstro commented Jul 6, 2024

jaimergp commented Jul 7, 2024

jaimergp commented Jul 8, 2024

hmaarrfk commented Jul 8, 2024