Basic GraalPy Support

[timfel]

With these changes, a GraalPy master build, and using maturin master, I can run the pytests and the examples. I have 3 failures in the pytests still. Please let me know if this is going in the right direction for you.

[adamreichold]

I think the most important question to clarify here is the status of this integration in the CI. If we are planning on proper support we should include it both for checking PR and for full target checks for the merge queues.

Is this possible and could you sketch an approach here?

[timfel]

I think the most important question to clarify here is the status of this integration in the CI. If we are planning on proper support we should include it both for checking PR and for full target checks for the merge queues.
Is this possible and could you sketch an approach here?

There are some changes for this that are only in graalpy master. Our releases go into pyenv, and our next release that will include the relevant changes will be out in September. So one option is to then setup the CI action to add graalpy to the matrix, install it using pyenv and simply run nox -s test-py with it. We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.

[adamreichold]

install it using pyenv and simply run nox -s test-py with it

Ideally, this would use exactly the same CI jobs we use for CPython and PyPy or is there a technical limitation which makes this unreasonable?

We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.

I think this mainly depends on how urgent this is for you, if you'd like to have this in our main branch before your release, then integrating a nightly build would be a prerequisite. Otherwise, waiting for the next release appears preferable.

[timfel]

install it using pyenv and simply run nox -s test-py with it

Ideally, this would use exactly the same CI jobs we use for CPython and PyPy or is there a technical limitation which makes this unreasonable?

We would first have to get graalpy into github actions.

We can also at any time do nightly builds and we start by adding an action that just uses a well-known nightly build in CI, and we switch to a release build via pyenv once that is out.

I think this mainly depends on how urgent this is for you, if you'd like to have this in our main branch before your release, then integrating a nightly build would be a prerequisite. Otherwise, waiting for the next release appears preferable.

We would of course prefer to have this in before our next release, so that we can start running the tests of actual packages that use PyO3 in our CI and fix any additional issues that occur before the September release.

[timfel]

We are adding support for graalpy to the setup-python action, then running it in the PyO3 CI should need minimal changes

[timfel]

The setup-python action maintainers had a question about GraalPy's licence, I hope they are happy with it soon and will merge, at which point I will rebase this PR and add GraalPy pytest to your CI (actions/setup-python#694)

[timfel]

This now passes all pytests on GraalPy 23.1 with the latest maturin release. Once PyO3/maturin#1773 and actions/setup-python#694 are merged, I can add GraalPy to the CI config.

[davidhewitt]

Great! Do you have a sense of how close the setup-python PR is to merging? I think we're going to push an 0.20 release imminently, so if this is extremely close we could wait for it. (Otherwise this could land in 0.20.1 later.)

[timfel]

Great! Do you have a sense of how close the setup-python PR is to merging? I think we're going to push an 0.20 release imminently, so if this is extremely close we could wait for it. (Otherwise this could land in 0.20.1 later.)

I couldn't say. We've tried to be very quick to do any requested changes, but I assume the github folks are busy and they end up not getting back to us for a week or two each time they ask for something. So don't wait for us.

[davidhewitt]

Ah okay, frustrating!

[davidhewitt]

@timfel I added CI-build-full label, so if you fix the fit issues and push again you should find we do a full CI run including the new graalpy job.

[timfel]

Ok, it's basically working now. The failure on python3.9 is a 500 server error downloading the manylinux image, so I suspect it's transient.

For GraalPy, we need a new release of maturin with this PR: PyO3/maturin#1802. I ran the CI config locally and with an updated maturin it passes that one step that's not running.

[davidhewitt]

Great, I will try to give this a full review in the coming days! I just got back from EuroRust with a ton of threads to follow up before they go cold as well as some time due for my family, so please forgive me if it takes a little longer than usual 🙏

[davidhewitt]

Thank you! Overall it looks very reasonable to me. Just a couple of style nits and a few questions to understand the future support and testing strategies for this.

@messense, when might you plan to do the next maturin release?

[timfel]

Do GraalPy wheels get uploaded to PyPI? Does the GraalPy version (presumably 24.0) go in the tag?

Yes, the graalpy and implemented cpython version go in the tag. pip debug gives wheel tags like graalpy310-graalpy240_310_native-manylinux_2_35_x86_64

[timfel]

I'm not entirely sure how the performance differences (both regressions and improvements) are caused by my changes. Maybe those benchmarks are so micro that there is jitter?

[davidhewitt]

Yep there's a few benchmarks which need to have volatility ignored after I re-enabled them in #3986

I cleared those now 👍

[timfel]

Is the failure in "CI / python3.12-x64 ubuntu-latest rust-nightly" something I could have done? It fails in pyo3-macros-backend, I haven't touched that, but I fear I don't understand enough to rule out some interaction with my changes.

[davidhewitt]

Nightly probably has improved dead code detection. I don't think you need to worry about it here.

[davidhewitt]

My view here is that we're now in agreement that this is suitable to merge in the current form, and there will be follow ups to tidy up various pieces subject to future work in graalpy.

I think given the size of the effort put in here it makes sense to see this merge now and release in 0.21, both to unblock the ecosystem and so that we can start learning about user experiences with this.

Thanks so much @timfel and @adamreichold for all the time invested here 👍