Pyodide improvements: version setting, standalone environments

[agriyakhetarpal]

Description

This PR updates the Pyodide build procedure (see #1456) that is enabled with CIBW_PLATFORM: "pyodide" (or with the --platform pyodide CLI equivalent) post the changes in pyodide/pyodide#4882, where pyodide/pyodide-build was unvendored from the main Pyodide repository to accommodate faster updates and fixes.

This means that the Pyodide version and pyodide-build are not going to be in sync going forward, and that the Pyodide xbuildenv to install must be inferred by the versions available to install by pyodide-build through a recently added pyodide xbuildenv search command, which prints out this table:

Tap to expand table

┌────────────┬────────────┬────────────┬───────────────────────────┬────────────┐
│ Version    │ Python     │ Emscripten │ pyodide-build             │ Compatible │
├────────────┼────────────┼────────────┼───────────────────────────┼────────────┤
│ 0.27.0a2   │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.4     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.3     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.2     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.1     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.0     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
└────────────┴────────────┴────────────┴───────────────────────────┴────────────┘

Alternatively, one may use pyodide xbuildenv search --all to return both compatible and non-compatible versions. This would, however, be better received as JSON (please see pyodide/pyodide-build#28).

---

Additionally, in this PR, support has been added for installing arbitrary Pyodide versions, or, more specifically, arbitrary versions for "Pyodide cross-build environments (xbuildenvs)" – though, only the ones that are supported for a given pyodide-build version. This has been implemented through an environment variable CIBW_PYODIDE_VERSION and an associated configuration variable in the schema (through a table implemented via pyodide/pyodide-build#26).

This PR uses the https://github.com/astral-sh/python-build-standalone project to retrieve runnable Python builds, so that we don't fall into compatibility issues where pyodide-build doesn't support a particular Python version. We fetch the appropriate version of Python required by the xbuildenv and pyodide-build and recommend users to check it out using the pyodide xbuildenv search command as needed.

The rationale behind this is that WebAssembly/Pyodide builds are already experimental, and it would be useful not to tie the available Pyodide version to the cibuildwheel version – this would be helpful for downstream projects (statsmodels/statsmodels#9343, scikit-image/scikit-image#7525, scikit-learn/scikit-learn#29791, and so on) to allow testing against Pyodide's alpha releases and/or for the case of greater reproducibility against Pyodide's older releases.

cc: @hoodmane and @ryanking13 for visibility

---

Suggested CHANGELOG entry

- ✨ Allow the use of a custom Pyodide version to target by the use of the `CIBW_PYODIDE_VERSION` environment variable or the `pyodide-version` configuration option. This is an experimental feature and users are advised to look at the [compatible Pyodide versions](https://github.com/pyodide/pyodide/blob/main/pyodide-cross-build-environments.json) according to the [`pyodide-build`](https://github.com/pyodide/pyodide-build) version, or install `pyodide-build` and run the `pyodide xbuildenv search --all` command to view a compatibility table.

[agriyakhetarpal]

The Windows test failures are unrelated. I'll try to fix them later in the day, but happy to step back if someone else does it before me, or wishes to.

[agriyakhetarpal]

Apologies from my end that it took so long, too!

[agriyakhetarpal]

Re the failing Windows tests; I guess we should make the tests in test_download.py also go through the cached GitHub API requests. I'll push a commit to do this.

The download function is not just for GitHub requests alone; never mind. Would it be a good idea to mock urllib.request.urlopen() for these tests?

[joerick]

Well, I'll have to eat my hat regarding githubusercontent URLs! Let's change the URL on that test to something more appropriate to a simple download test.

Perhaps https://badssl.com/ would work.

[agriyakhetarpal]

Argh, perhaps it's a good cue to revisit Statically? There is also GitHack: https://raw.githack.com/. The point about security concerns and supply-chain attacks still exists, though. The uptime is quite appealing: https://status.statically.io/

Another idea: if we set the GitHub Actions' standard GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} as an environment variable in the workflow file, and detect it during the relevant tests using os.environ.get("GITHUB_TOKEN", ""), maybe it'll get propagated and we won't hit API rate limits that easily?

[joerick]

Relying on the contents of that HTML file at badssl.com to not change doesn't feel like a good test assertion. I've switched to jsdelivr for that one.

For get_pyodide_xbuildenv_info() I've stuck with the GitHub URL, because the CDNs will cache (e.g. for jsdelivr, it's 12-24 hours) and we want up-to-date versions when rev'ing dependencies.

I'll raise an issue on pyodide-build regarding the HTTP 429 seen there.

[agriyakhetarpal]

I'll mark this as a draft until pyodide/pyodide-build#203 and pyodide/pyodide#5642 are resolved.

[henryiii]

This is passing, what's the status? :)

[agriyakhetarpal]

This is passing, what's the status? :)

It's ready to go in now! I released pyodide/pyodide-build@0.30.4 (release) with the new cross-build environment metadata URL and updated the constraints.

[agriyakhetarpal]

The ubuntu-py3.14 failure with Pyodide is unrelated; it came from a GitHub rate limit affecting _ensure_virtualenv. :( We should still be good to go, I'll re-trigger CI. :)

[agriyakhetarpal]

Surfacing this conversation as there are a lot of them getting hidden in the UI: #2002 (comment)