Skip to content

Commit 1638641

Browse files
agriyakhetarpalhoodmanejoerickhenryiiipre-commit-ci[bot]
authored
feat: Pyodide improvements: version setting, standalone environments (#2002)
* Fix a typo: pyoodide ➡️ pyodide * Add `pyodide_build_version` attribute * Add version to xbuildenv log step * Add version to Emscripten log step * Use `pyodide-build`'s version for updating constraints * Bump Pyodide constraints by updating `pyodide-build` * Add a schema for `pyodide-version` * Update Pyodide constraints * Bump `pyodide-build` to new 0.29.0 * Test out another Pyodide identifier * Update outdated Pyodide constraints * Add Pyodide version to temp directory name * Remove Pyodide 0.26.1 from build configurations * Retrieve + validate + install specific xbuildenvs * Test wheel builds with Pyodide 0.26.2 * Add correct Pyodide version to identifier temp dir * Don't pre-call Pyodide xbuildenv search * Fetch just the stable Pyodide versions * Refactor search + validation + install into one step * Move all of it under a lock * Reorder xbuildenv installation * Add env and cwd to xbuildenv search call * Temporarily lower to 0.26.2 target * Separate out search, validate, install; again * Run xbuildenv search in `CIBW_CACHE_PATH` * Remove prior `PYODIDE_ROOT` env vars, copy envs * Validate doesn't need to depend on searching * Add file lock when searching xbuildenvs * Test the original version: 0.26.1 * Update Pyodide constraints * Update constraints for `pyodide-build` 0.29.0 again * Bump Pyodide from version 0.26.1 ➡️ version 0.26.4 * Add note on compatibility for macOS + other archs * Note Pyodide version for Pyodide identifier * Docs about `CIBW_PYODIDE_VERSION` * Don't fetch just the stable versions * Discard a variable that's not used later * Rename `search_xbuildenv` ➡️ `get_xbuildenv_versions` * `validate_xbuildenv` ➡️ `validate_xbuildenv_version` * Replace ordered comment, add newline * Replace sentence on macOS support Co-Authored-By: Hood Chatham <[email protected]> * Capitalise: "pyodide" ➡️ "Pyodide" Co-Authored-By: Hood Chatham <[email protected]> * "work" ➡️ "may succeed" Co-Authored-By: Hood Chatham <[email protected]> * Add another job to test a custom Pyodide version * Handle "v"-prefixed + non-prefixed versions * Convert to a proper toml-able option, and remove some hardcoded versions This removes the enscripten and pyodide-build version specs from pyproject.toml - pyodide-build is spec'd in the constraints file, and the emscripten version can be read from the pyodide-build output. * Add a schema entry * Add docs for CIBW_PYODIDE_VERSION * Rephrase * Add tests for pyodide-version * Apply suggestions from code review * Add python_build_standalone util * Hook up to python-build-standalone, removing dependency on host python * Remove python hard-code in action.yml * Add log step * Add workaround for pyodide/pyodide-build#143 * Add emscripten pytest test * Remove unneeded checks * Fix a pytest invoke for emscripten * Generate pyodide-build constraints from the pinned pyodide version * Remove pyodide python-build-standalone workaround * Fixup paths from newer version of pyodide-build * Fix/skip some failing tests * Use `python -m pytest` on pyodide, even on Linux * Docs fixes * Don't call the github API at runtime, cache the release assets instead * Ignore pylint false positive * Add version auto-updating for pyodide * Add support for pyodide 3.13. * Fix tests for multiple pyodide wheels * Remove workaround for unreleased pyodide-build * Rename to "test_pyodide" * Fix pathname confusion * Remove extra github actions job * Fix expectation for test_abi_none * Fix the custom_repair_wheel test to actually have clashing names * Fix pinned version test * Document test-command limitation * Remove pyodide 0.28.0a1 for now * Update constraints files * Docs/test fixes post removing pyodide cp313 * Fix ABI test expectation * Docs improvements * Improve some comments * Remove logic duplication * remove pyodide special casing * Refactor constraints code to use a utility script, circumventing import issues * chore: nicer nox env Signed-off-by: Henry Schreiner <[email protected]> * fix: typo in variable name found by copilot Signed-off-by: Henry Schreiner <[email protected]> * Apply suggestions from code review * Some more Pyodide version updates in the docs section * We haven't released Pyodide v0.27.6 yet * Back to the Github URL for cross-build-environments * `pyodide-build`, not `emsdk` for Windows skips Co-Authored-By: Hood Chatham <[email protected]> * Move to a separate `_json_request` function Co-Authored-By: Hood Chatham <[email protected]> * Rename "retries" ➡️ "retry_count" * Add some type hints * Copy env vars before `UV_CUSTOM_COMPILE_COMMAND` * Use `HTTPError.headers.get` instead * Remove extra end quote * Change download tests URL to `https://badssl.com/` Co-Authored-By: Joe Rickerby <[email protected]> * Download size changes, too * Use jsdelivr for Github asset mirroring * Bump to Pyodide v0.27.6 * Fix unit tests * Move to pyodide v0.27.6 again * Bump to pyodide-build 0.30.4 * Use new URL for cross-build environments metadata Co-authored-by: Joe Rickerby <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Henry Schreiner <[email protected]> Co-authored-by: Hood Chatham <[email protected]> Co-authored-by: Joe Rickerby <[email protected]> Co-authored-by: Henry Schreiner <[email protected]> Co-authored-by: Joe Rickerby <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent c3c260d commit 1638641

30 files changed

+1298
-115
lines changed

.github/workflows/test.yml

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -195,7 +195,7 @@ jobs:
195195
run: uv run --no-sync pytest --run-emulation ${{ matrix.arch }} test/test_emulation.py
196196

197197
test-pyodide:
198-
name: Test cibuildwheel building Pyodide wheels
198+
name: Test pyodide
199199
needs: lint
200200
runs-on: ubuntu-24.04
201201
timeout-minutes: 180
@@ -222,6 +222,19 @@ jobs:
222222
env:
223223
CIBW_PLATFORM: pyodide
224224

225+
- name: Run a sample build (GitHub Action) for an overridden Pyodide version
226+
uses: ./
227+
with:
228+
package-dir: sample_proj
229+
output-dir: wheelhouse
230+
# In case this breaks at any point in time, switch to using the latest version
231+
# available or any other version that is not the same as the default one set
232+
# in cibuildwheel/resources/build-platforms.toml.
233+
env:
234+
CIBW_PLATFORM: pyodide
235+
CIBW_BUILD: "cp312*"
236+
CIBW_PYODIDE_VERSION: "0.27.6"
237+
225238
- name: Run tests with 'CIBW_PLATFORM' set to 'pyodide'
226239
run: |
227240
uv run --no-sync ./bin/run_tests.py

action.yml

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,10 @@ branding:
2424
runs:
2525
using: composite
2626
steps:
27-
# Set up the version of Python that supports pyodide
2827
- uses: actions/setup-python@v5
2928
id: python
3029
with:
31-
python-version: "3.12"
30+
python-version: "3.11 - 3.13"
3231
update-environment: false
3332

3433
- id: cibw

bin/generate_pyodide_constraints.py

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
#!/usr/bin/env python3
2+
3+
import sys
4+
import textwrap
5+
from pathlib import Path
6+
7+
import click
8+
9+
from cibuildwheel.extra import get_pyodide_xbuildenv_info
10+
11+
12+
@click.command()
13+
@click.argument(
14+
"pyodide-version",
15+
type=str,
16+
)
17+
@click.option(
18+
"--output-file",
19+
type=click.Path(),
20+
default=None,
21+
help="Output file to write the constraints to. If not provided, the constraints will be printed to stdout.",
22+
)
23+
def generate_pyodide_constraints(pyodide_version: str, output_file: str | None = None) -> None:
24+
"""
25+
Generate constraints for a specific Pyodide version. The constraints are
26+
generated based on the Pyodide version's xbuildenv info, which is retrieved
27+
from the Pyodide repository.
28+
29+
These constraints should then be 'pinned' using `uv pip compile`.
30+
31+
Example usage:
32+
33+
bin/generate_pyodide_constraints.py 0.27.0
34+
"""
35+
xbuildenv_info = get_pyodide_xbuildenv_info()
36+
try:
37+
pyodide_version_xbuildenv_info = xbuildenv_info["releases"][pyodide_version]
38+
except KeyError as e:
39+
msg = f"Pyodide version {pyodide_version} not found in xbuildenv info. Versions available: {', '.join(xbuildenv_info['releases'].keys())}"
40+
raise click.BadParameter(msg) from e
41+
42+
pyodide_build_min_version = pyodide_version_xbuildenv_info.get("min_pyodide_build_version")
43+
pyodide_build_max_version = pyodide_version_xbuildenv_info.get("max_pyodide_build_version")
44+
45+
pyodide_build_specifier_parts: list[str] = []
46+
47+
if pyodide_build_min_version:
48+
pyodide_build_specifier_parts.append(f">={pyodide_build_min_version}")
49+
if pyodide_build_max_version:
50+
pyodide_build_specifier_parts.append(f"<={pyodide_build_max_version}")
51+
52+
pyodide_build_specifier = ",".join(pyodide_build_specifier_parts)
53+
54+
constraints_txt = textwrap.dedent(f"""
55+
pip
56+
build[virtualenv]
57+
pyodide-build{pyodide_build_specifier}
58+
click<8.2
59+
""")
60+
61+
if output_file is None:
62+
print(constraints_txt)
63+
else:
64+
Path(output_file).write_text(constraints_txt)
65+
print(f"Constraints written to {output_file}", file=sys.stderr)
66+
67+
68+
if __name__ == "__main__":
69+
generate_pyodide_constraints()

bin/generate_schema.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,9 @@
197197
xbuild-tools:
198198
description: Binaries on the path that should be included in an isolated cross-build environment
199199
type: string_array
200+
pyodide-version:
201+
type: string
202+
description: Specify the version of Pyodide to use
200203
repair-wheel-command:
201204
description: Execute a shell command to repair each built wheel.
202205
type: string_array

bin/update_python_build_standalone.py

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/usr/bin/env python3
2+
3+
import json
4+
5+
from cibuildwheel.extra import github_api_request
6+
from cibuildwheel.util.python_build_standalone import (
7+
PythonBuildStandaloneAsset,
8+
PythonBuildStandaloneReleaseData,
9+
)
10+
from cibuildwheel.util.resources import PYTHON_BUILD_STANDALONE_RELEASES
11+
12+
13+
def main() -> None:
14+
"""
15+
This script updates the vendored list of release assets to the latest
16+
version of astral-sh/python-build-standalone.
17+
"""
18+
19+
# Get the latest release tag from the GitHub API
20+
latest_release = github_api_request("repos/astral-sh/python-build-standalone/releases/latest")
21+
latest_tag = latest_release["tag_name"]
22+
23+
# Get the list of assets for the latest release
24+
github_assets = github_api_request(
25+
f"repos/astral-sh/python-build-standalone/releases/tags/{latest_tag}"
26+
)["assets"]
27+
28+
assets: list[PythonBuildStandaloneAsset] = []
29+
30+
for github_asset in github_assets:
31+
name = github_asset["name"]
32+
if not name.endswith("install_only.tar.gz"):
33+
continue
34+
url = github_asset["browser_download_url"]
35+
assets.append({"name": name, "url": url})
36+
37+
# Write the assets to the JSON file. One day, we might need to support
38+
# multiple releases, but for now, we only support the latest one
39+
json_file_contents: PythonBuildStandaloneReleaseData = {
40+
"releases": [
41+
{
42+
"tag": latest_tag,
43+
"assets": assets,
44+
}
45+
]
46+
}
47+
48+
with PYTHON_BUILD_STANDALONE_RELEASES.open("w", encoding="utf-8") as f:
49+
json.dump(json_file_contents, f, indent=2)
50+
# Add a trailing newline, our pre-commit hook requires it
51+
f.write("\n")
52+
53+
print(
54+
f"Updated {PYTHON_BUILD_STANDALONE_RELEASES.name} with {len(assets)} assets for tag {latest_tag}"
55+
)
56+
57+
58+
if __name__ == "__main__":
59+
main()

bin/update_pythons.py

Lines changed: 51 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
from rich.logging import RichHandler
2020
from rich.syntax import Syntax
2121

22-
from cibuildwheel.extra import dump_python_configurations
22+
from cibuildwheel.extra import dump_python_configurations, get_pyodide_xbuildenv_info
2323

2424
log = logging.getLogger("cibw")
2525

@@ -57,7 +57,14 @@ class ConfigApple(TypedDict):
5757
url: str
5858

5959

60-
AnyConfig = ConfigWinCP | ConfigWinPP | ConfigWinGP | ConfigApple
60+
class ConfigPyodide(TypedDict):
61+
identifier: str
62+
version: str
63+
default_pyodide_version: str
64+
node_version: str
65+
66+
67+
AnyConfig = ConfigWinCP | ConfigWinPP | ConfigWinGP | ConfigApple | ConfigPyodide
6168

6269

6370
# The following set of "Versions" classes allow the initial call to the APIs to
@@ -347,6 +354,39 @@ def update_version_ios(self, identifier: str, version: Version) -> ConfigApple |
347354
return None
348355

349356

357+
class PyodideVersions:
358+
def __init__(self) -> None:
359+
xbuildenv_info = get_pyodide_xbuildenv_info()
360+
self.releases = xbuildenv_info["releases"]
361+
362+
def update_version_pyodide(
363+
self, identifier: str, version: Version, spec: Specifier, node_version: str
364+
) -> ConfigPyodide | None:
365+
# get releases that match the python version
366+
releases = [
367+
r for r in self.releases.values() if spec.contains(Version(r["python_version"]))
368+
]
369+
# sort by version, latest first
370+
releases.sort(key=lambda r: Version(r["version"]), reverse=True)
371+
372+
if not releases:
373+
msg = f"Pyodide not found for {spec}!"
374+
raise ValueError(msg)
375+
376+
final_releases = [r for r in releases if not Version(r["version"]).is_prerelease]
377+
378+
# prefer a final release if available, otherwise use the latest
379+
# pre-release
380+
release = final_releases[0] if final_releases else releases[0]
381+
382+
return ConfigPyodide(
383+
identifier=identifier,
384+
version=str(version),
385+
default_pyodide_version=release["version"],
386+
node_version=node_version,
387+
)
388+
389+
350390
# This is a universal interface to all the above Versions classes. Given an
351391
# identifier, it updates a config dict.
352392

@@ -369,6 +409,8 @@ def __init__(self) -> None:
369409

370410
self.graalpy = GraalPyVersions()
371411

412+
self.pyodide = PyodideVersions()
413+
372414
def update_config(self, config: MutableMapping[str, str]) -> None:
373415
identifier = config["identifier"]
374416
version = Version(config["version"])
@@ -407,6 +449,10 @@ def update_config(self, config: MutableMapping[str, str]) -> None:
407449
config_update = self.windows_arm64.update_version_windows(spec)
408450
elif "ios" in identifier:
409451
config_update = self.ios_cpython.update_version_ios(identifier, version)
452+
elif "pyodide" in identifier:
453+
config_update = self.pyodide.update_version_pyodide(
454+
identifier, version, spec, config["node_version"]
455+
)
410456

411457
assert config_update is not None, f"{identifier} not found!"
412458
config.update(**config_update)
@@ -445,6 +491,9 @@ def update_pythons(force: bool, level: str) -> None:
445491
for config in configs["ios"]["python_configurations"]:
446492
all_versions.update_config(config)
447493

494+
for config in configs["pyodide"]["python_configurations"]:
495+
all_versions.update_config(config)
496+
448497
result_toml = dump_python_configurations(configs)
449498

450499
rich.print() # spacer

cibuildwheel/extra.py

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,16 @@
22
These are utilities for the `/bin` scripts, not for the `cibuildwheel` program.
33
"""
44

5+
import json
6+
import time
7+
import typing
8+
import urllib.error
9+
import urllib.request
510
from collections.abc import Mapping, Sequence
611
from io import StringIO
7-
from typing import Protocol
12+
from typing import Any, NotRequired, Protocol
13+
14+
from cibuildwheel import __version__ as cibw_version
815

916
__all__ = ("Printable", "dump_python_configurations")
1017

@@ -30,3 +37,65 @@ def dump_python_configurations(
3037
output.write("\n")
3138
# Strip the final newline, to avoid two blank lines at the end.
3239
return output.getvalue()[:-1]
40+
41+
42+
def _json_request(request: urllib.request.Request, timeout: int = 30) -> dict[str, Any]:
43+
with urllib.request.urlopen(request, timeout=timeout) as response:
44+
return typing.cast(dict[str, Any], json.load(response))
45+
46+
47+
def github_api_request(path: str, *, max_retries: int = 3) -> dict[str, Any]:
48+
"""
49+
Makes a GitHub API request to the given path and returns the JSON response.
50+
"""
51+
api_url = f"https://api.github.com/{path}"
52+
headers = {
53+
"Accept": "application/vnd.github.v3+json",
54+
"User-Agent": f"cibuildwheel/{cibw_version}",
55+
}
56+
request = urllib.request.Request(api_url, headers=headers)
57+
58+
for retry_count in range(max_retries):
59+
try:
60+
return _json_request(request)
61+
except (urllib.error.URLError, TimeoutError) as e:
62+
# pylint: disable=E1101
63+
if (
64+
isinstance(e, urllib.error.HTTPError)
65+
and (e.code == 403 or e.code == 429)
66+
and e.headers.get("x-ratelimit-remaining") == "0"
67+
):
68+
reset_time = int(e.headers.get("x-ratelimit-reset", 0))
69+
wait_time = max(0, reset_time - int(e.headers.get("date", 0)))
70+
print(f"Github rate limit exceeded. Waiting for {wait_time} seconds.")
71+
time.sleep(wait_time)
72+
else:
73+
print(f"Retrying GitHub API request due to error: {e}")
74+
75+
if retry_count == max_retries - 1:
76+
print(f"GitHub API request failed (Network error: {e}). Check network connection.")
77+
raise e
78+
79+
# Should never be reached but to keep the type checker happy
80+
msg = "Unexpected execution path in github_api_request"
81+
raise RuntimeError(msg)
82+
83+
84+
class PyodideXBuildEnvRelease(typing.TypedDict):
85+
version: str
86+
python_version: str
87+
emscripten_version: str
88+
min_pyodide_build_version: NotRequired[str]
89+
max_pyodide_build_version: NotRequired[str]
90+
91+
92+
class PyodideXBuildEnvInfo(typing.TypedDict):
93+
releases: dict[str, PyodideXBuildEnvRelease]
94+
95+
96+
def get_pyodide_xbuildenv_info() -> PyodideXBuildEnvInfo:
97+
xbuildenv_info_url = (
98+
"https://pyodide.github.io/pyodide/api/pyodide-cross-build-environments.json"
99+
)
100+
with urllib.request.urlopen(xbuildenv_info_url) as response:
101+
return typing.cast(PyodideXBuildEnvInfo, json.loads(response.read().decode("utf-8")))

cibuildwheel/options.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,7 @@ class BuildOptions:
112112
build_frontend: BuildFrontendConfig | None
113113
config_settings: str
114114
container_engine: OCIContainerEngineConfig
115+
pyodide_version: str | None
115116

116117
@property
117118
def package_dir(self) -> Path:
@@ -850,6 +851,8 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
850851
msg = f"Failed to parse container config. {e}"
851852
raise errors.ConfigurationError(msg) from e
852853

854+
pyodide_version = self.reader.get("pyodide-version", env_plat=False)
855+
853856
return BuildOptions(
854857
globals=self.globals,
855858
test_command=test_command,
@@ -871,6 +874,7 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
871874
build_frontend=build_frontend,
872875
config_settings=config_settings,
873876
container_engine=container_engine,
877+
pyodide_version=pyodide_version or None,
874878
)
875879

876880
def check_for_invalid_configuration(self, identifiers: Iterable[str]) -> None:

0 commit comments

Comments
 (0)