Add support for building Android wheels #2349

mhsmith · 2025-04-04T23:28:20Z

Closes Add support for Android and iOS #1960

This PR depends on a number of changes to CPython's android.py script, which are being developed in python/cpython#132870.

… Python installation

henryiii · 2025-04-23T13:35:25Z

Let us know if you need (the rest of) CI triggered. :)

…iables

mhsmith · 2025-05-04T23:08:41Z

@henryiii: This is close to being complete, so please enable the rest of CI.

mhsmith · 2025-05-06T20:33:42Z

To run the integration tests on most of the CI machines, it looks like I'll need to automate installation of the correct Python version. For macOS this can be done the same way as iOS, but for Linux there's no existing code to reuse, because the native Linux build uses Docker. So I'll probably implement something that uses python-build-standalone, unless anyone has another suggestion.

Apart from that, I think this PR is complete enough now that it's worth reviewing. @freakboy3742 and anyone else who's interested.

henryiii · 2025-05-06T20:53:35Z

python-build-standalone is fine, in fact, I'd like to use that for pyodide in the future, too.

joerick · 2025-05-07T10:50:09Z

I've already written code to install a version of python-build-standalone in #2002, along with version pinning etc. I think it would work nicely here too.

freakboy3742

Comments inline; my two high level concerns are:

Whether the Builder class actually delivers any benefit here; and
The general approach around avoiding python -m build in order to get platform-specific build requirements.

freakboy3742 · 2025-05-08T01:52:32Z

cibuildwheel/resources/_cross_venv.py

+
+    # platform ################################################################
+    #
+    # We can't determine the user-visible Android version number from the API level, so return a


Any need to patch platform.uname() and platform.system()?

It's enough to patch sys.platform and platform.android_ver, and the other platform functions will then do the right thing.

freakboy3742 · 2025-05-08T01:55:16Z

cibuildwheel/resources/_cross_venv.py

+
+    # sysconfig ###############################################################
+    #
+    sysconfig._init_config_vars()  # type: ignore[attr-defined]


Does this generate the right values in sysconfig.get_paths()? On iOS, additional handling is needed to ensure that include doesn't point at the build install.

Also - does subprocess._can_fork_exec need patching?

The sysconfig paths are handled in localize_sysconfigdata in android.py. We alter the sysconfigdata file before the pth file even executes.

subprocess._can_fork_exec doesn't need patching, because unlike on iOS, subprocesses are possible on Android, they're just not recommended. So _can_fork_exec is already true.

freakboy3742 · 2025-05-08T01:57:56Z

docs/platforms.md

+
+### Build frontend support
+
+Android builds only support the `build` frontend. In principle, support for the


Any reason there's no support for the pip frontend?

The build frontend does two things:

Create the build environment.

Call the PEP 517 "build wheel" API within that environment.

In this case cibuildwheel is creating the build environment itself, so the only thing we're using the build frontend for is to convert a command line into a PEP 517 API call. pip and build should produce identical calls here, so there's no point in doubling the amount of code and testing.

There's some precedent for this, as pyodide also requires the build frontend. build has also now become the default frontend in cibuildwheel 3.0.

pyproject.toml

freakboy3742 · 2025-05-08T02:04:43Z

test/test_android.py

+
+system_machine = (platform.system(), platform.machine())
+if system_machine not in [("Linux", "x86_64"), ("Darwin", "arm64"), ("Darwin", "x86_64")]:
+    pytest.skip(


Should this skip if ANDROID_HOME isn't defined (or some other marker that Android tools are available?) Not sure if the preference should be to skip or fail if you're on a platform that could run Android tests, but you don't have an Android environment configured - or if that's something where it's an xfail locally, but a hard fail in an actual CI environment.

At the very least, there's an option to fail this module fast and early if ANDROID_HOME doesn't exist, rather than trying to run a handful of tests that we can reliably predict won't work.

Good point, I'll just make it skip.

freakboy3742 · 2025-05-08T03:13:32Z

cibuildwheel/platforms/android.py

+            "--extra-index-url",
+            "https://chaquo.com/pypi-13.1/",


Why is this needed? Won't it potentially introduce a dependency on "non-official" binary builds?

freakboy3742 · 2025-05-08T03:16:13Z

docs/options.md

+
+* `python -c command ...`
+* `python -m module_name ...`
+* `pytest ...` (converted to `python -m pytest ...`)


Flagging the obvious overlap here with #2363.

freakboy3742 · 2025-05-08T03:18:23Z

cibuildwheel/platforms/android.py

+        shutil.rmtree(self.tmp_dir)
+        log.build_end()
+
+    def setup_python(self) -> None:


Flagging that this should probably differentiate setup_target_python() from setup_build_python() if a dependency on standalone is added.

freakboy3742 · 2025-05-08T03:20:10Z

cibuildwheel/platforms/android.py

+        # Apply custom environment variables, and check environment is still valid
+        self.env = self.build_options.environment.as_dictionary(self.env)
+        self.env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
+        for command in ["python", "pip"]:


Is this not duplicating the check for python a few lines above (139-143)?

This is idiomatic in cibuildwheel, the second check is to make sure that the user didn't mess up the PATH variable in CIBW_ENVIRONMENT by doing something like PATH=/usr/local/bin:$PATH and thus changing the active python interpreter.

freakboy3742 · 2025-05-08T03:26:21Z

cibuildwheel/platforms/android.py

+        self.pip_install("build", *constraint_flags(dependency_constraint))
+
+        # Install build-time requirements. These must be installed for the build platform, not for
+        # Android, which is why we can't allow them to be installed by the `build` subprocess.


So - I know the context and background because we've spoken about this in person; but the general approach is a bit weird (and different to iOS). It would be worth giving a high-level description of what is going on here - that it's deliberately not invoking python -m build, and instead reproducing what build does in steps so that the build dependencies can be explicitly installed in an "non-cross" environment.

I guess the bigger question is whether this is something we should be doing at all - the iOS example shows that some packages can be compiled without this workaround; and there's clearly a bigger discussion that needs to happen around build and the PEP 517 interface.

joerick · 2025-05-09T10:50:31Z

cibuildwheel/architecture.py

+def native_platform() -> PlatformName:
+    if sys.platform.startswith("linux"):
+        return "linux"
+    elif sys.platform == "darwin":
+        return "macos"
+    elif sys.platform == "win32":
+        return "windows"
+    else:
+        msg = (
+            'Unable to detect platform from "sys.platform". cibuildwheel doesn\'t '
+            "support building wheels for this platform. You might be able to build for a different "
+            "platform using the --platform argument. Check --help output for more information."
+        )
+        raise errors.ConfigurationError(msg)
+
+


could we move this to platform/__init__.py? or does that hit a circular import problem...

joerick · 2025-05-09T11:25:51Z

cibuildwheel/frontend.py

+        if existing_value is None:
+            config_settings[setting] = value
+        elif isinstance(existing_value, str):
+            config_settings[setting] = [existing_value]


it looks to me that value would be discarded in this branch?

Suggested change

config_settings[setting] = [existing_value]

config_settings[setting] = [existing_value, value]

joerick · 2025-05-09T11:34:56Z

cibuildwheel/platforms/android.py

+
+
+@dataclass
+class Builder:


I'd say +1, for the sake of consistency with the other platforms, and passing around lots of variables (while it's a bit of a drag) prevents other classes of bugs in its explicitness.

joerick · 2025-05-09T11:36:27Z

cibuildwheel/platforms/android.py

+    ]
+
+
+def shell_prepared(command: str, build_options: BuildOptions, env: dict[str, str]) -> None:


Could we add some kwargs, for the sake of readability?

Suggested change

def shell_prepared(command: str, build_options: BuildOptions, env: dict[str, str]) -> None:

def shell_prepared(command: str, *, build_options: BuildOptions, env: dict[str, str]) -> None:

joerick · 2025-05-09T11:41:02Z

cibuildwheel/platforms/android.py

+        # Apply custom environment variables, and check environment is still valid
+        self.env = self.build_options.environment.as_dictionary(self.env)
+        self.env["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
+        for command in ["python", "pip"]:


This is idiomatic in cibuildwheel, the second check is to make sure that the user didn't mess up the PATH variable in CIBW_ENVIRONMENT by doing something like PATH=/usr/local/bin:$PATH and thus changing the active python interpreter.

joerick · 2025-05-09T18:42:40Z

cibuildwheel/platforms/android.py

+        for path in [
+            resources.PATH / "_cross_venv.py",
+            next(self.python_dir.glob("prefix/lib/python*/_sysconfigdata_*.py")),
+        ]:
+            out_path = Path(shutil.copy(path, self.site_packages))
+            if "sysconfigdata" in path.name:
+                self.localize_sysconfigdata(out_path)


Can we unroll this loop? I was very confused why these files were in a list together.

joerick · 2025-05-09T18:46:29Z

cibuildwheel/platforms/android.py

+    def pip_install(self, *args: PathOrStr) -> None:
+        if args:
+            call("pip", "install", "--upgrade", *args, env=self.env)


This feels like an unnecessary abstraction imo. Could we inline this please?

joerick · 2025-05-09T18:49:31Z

cibuildwheel/platforms/android.py

+            env_output = call(
+                self.python_dir / "android.py", "env", env=self.env, capture_stdout=True
+            )


Where does android.py come from? I don't see it get created.

It's part of cpython, and included as part of an "Python on Android binary release".

joerick · 2025-05-09T18:54:54Z

cibuildwheel/platforms/android.py

+        os.environ.update(self.android_env)
+
+        pth_file = self.site_packages / "_cross_venv.pth"
+        pth_file.write_text("import _cross_venv; _cross_venv.initialize()")
+
+        try:
+            yield
+        finally:
+            pth_file.unlink()
+            for key, original_value in original_env.items():
+                if original_value is None:
+                    del os.environ[key]
+                else:
+                    os.environ[key] = original_value


Rather modifying global state here, it would be better to return (or yield, if necessary) an env dict here that contained the environment appropriate for cross-compilation.

pyproject.toml

mhsmith added 2 commits April 5, 2025 00:14

Add Android to resource files

3a3caec

Add Android to miscellaneous places

578c181

freakboy3742 mentioned this pull request Apr 7, 2025

Back port from flet-dev/mobile-forge beeware/mobile-forge#96

Open

mhsmith added 4 commits April 15, 2025 23:22

Add Android documentation

c720392

Merge remote-tracking branch 'origin/main' into android

e33235d

Docs cleanups

8e03e87

Add Android platform module; implement top-level structure and target…

4428868

… Python installation

Implement setup_env and build_wheel

f3084f0

mhsmith mentioned this pull request Apr 24, 2025

gh-131531: android.py enhancements to support cibuildwheel python/cpython#132870

Open

mhsmith added 2 commits April 25, 2025 19:08

lru-dict build working

e715aa4

Alter prefix in sysconfigdata file; fix various issues with FLAGS var…

4234013

…iables

mhsmith mentioned this pull request May 4, 2025

Change to the iOS testing option semantics #2363

Merged

mhsmith added 3 commits May 4, 2025 22:46

Implement Android testing

f2048b7

Merge remote-tracking branch 'origin/main' into android

ec1a6c5

Add type annotations to _cross_venv

be664be

mhsmith added 4 commits May 5, 2025 19:43

Revert Python 3.8 to pip 25.0.1

d56114b

Merge remote-tracking branch 'origin/main' into android

13003df

Make test-sources required on Android

71ba70d

Add Android integration tests

e1ef58f

mhsmith marked this pull request as ready for review May 6, 2025 20:21

mhsmith mentioned this pull request May 7, 2025

Importer doesn't accept Android SOABI suffix chaquo/chaquopy#1370

Closed

joerick mentioned this pull request May 7, 2025

Pyodide improvements: version setting, standalone environments #2002

Merged

freakboy3742 suggested changes May 8, 2025

View reviewed changes

joerick mentioned this pull request May 8, 2025

Release plans: 2.22 and 3.0 #2047

Open

joerick reviewed May 9, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for building Android wheels #2349

Add support for building Android wheels #2349

mhsmith commented Apr 4, 2025 •

edited

Loading

henryiii commented Apr 23, 2025

mhsmith commented May 4, 2025

mhsmith commented May 6, 2025

henryiii commented May 6, 2025

joerick commented May 7, 2025

freakboy3742 left a comment

freakboy3742 May 8, 2025

mhsmith May 14, 2025

freakboy3742 May 8, 2025

freakboy3742 May 8, 2025

mhsmith May 14, 2025

freakboy3742 May 8, 2025

mhsmith May 14, 2025

freakboy3742 May 8, 2025

mhsmith May 14, 2025

freakboy3742 May 8, 2025

freakboy3742 May 8, 2025

freakboy3742 May 8, 2025

freakboy3742 May 8, 2025

joerick May 9, 2025

freakboy3742 May 8, 2025

joerick May 9, 2025

joerick May 9, 2025

joerick May 9, 2025

joerick May 9, 2025

joerick May 9, 2025

joerick May 9, 2025

joerick May 9, 2025

joerick May 9, 2025

freakboy3742 May 10, 2025 •

edited

Loading

joerick May 9, 2025


		### Build frontend support

		Android builds only support the `build` frontend. In principle, support for the

	config_settings[setting] = [existing_value]
	config_settings[setting] = [existing_value, value]

		]


		def shell_prepared(command: str, build_options: BuildOptions, env: dict[str, str]) -> None:

	def shell_prepared(command: str, build_options: BuildOptions, env: dict[str, str]) -> None:
	def shell_prepared(command: str, *, build_options: BuildOptions, env: dict[str, str]) -> None:



		@dataclass
		class Builder:

Add support for building Android wheels #2349

Are you sure you want to change the base?

Add support for building Android wheels #2349

Conversation

mhsmith commented Apr 4, 2025 • edited Loading

henryiii commented Apr 23, 2025

mhsmith commented May 4, 2025

mhsmith commented May 6, 2025

henryiii commented May 6, 2025

joerick commented May 7, 2025

freakboy3742 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freakboy3742 May 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhsmith commented Apr 4, 2025 •

edited

Loading

freakboy3742 May 10, 2025 •

edited

Loading