Skip to content

Commit

Permalink
Merge branch 'main' into webbrowser-argparse
Browse files Browse the repository at this point in the history
  • Loading branch information
hugovk authored Mar 28, 2024
2 parents d502d2c + c1712ef commit a26c4b0
Show file tree
Hide file tree
Showing 445 changed files with 18,049 additions and 12,274 deletions.
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Python/dynload_*.c @ericsnowcurrently
Lib/test/test_module/ @ericsnowcurrently
Doc/c-api/module.rst @ericsnowcurrently
**/*importlib/resources/* @jaraco @warsaw @FFY00
**/importlib/metadata/* @jaraco @warsaw
**/*importlib/metadata/* @jaraco @warsaw

# Dates and times
**/*datetime* @pganssle @abalkin
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Lib/test/data/*
/_bootstrap_python
/Makefile
/Makefile.pre
iOSTestbed.*
/iOSTestbed.*
iOS/Frameworks/
iOS/Resources/Info.plist
iOS/testbed/build
Expand Down
4 changes: 3 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.2.0
rev: v0.3.4
hooks:
- id: ruff
name: Run Ruff on Lib/test/
Expand All @@ -14,6 +14,8 @@ repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: check-case-conflict
- id: check-merge-conflict
- id: check-toml
exclude: ^Lib/test/test_tomllib/
- id: check-yaml
Expand Down
64 changes: 64 additions & 0 deletions Android/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Python for Android

These instructions are only needed if you're planning to compile Python for
Android yourself. Most users should *not* need to do this. If you're looking to
use Python on Android, one of the following tools will provide a much more
approachable user experience:

* [Briefcase](https://briefcase.readthedocs.io), from the BeeWare project
* [Buildozer](https://buildozer.readthedocs.io), from the Kivy project
* [Chaquopy](https://chaquo.com/chaquopy/)


## Prerequisites

Export the `ANDROID_HOME` environment variable to point at your Android SDK. If
you don't already have the SDK, here's how to install it:

* Download the "Command line tools" from <https://developer.android.com/studio>.
* Create a directory `android-sdk/cmdline-tools`, and unzip the command line
tools package into it.
* Rename `android-sdk/cmdline-tools/cmdline-tools` to
`android-sdk/cmdline-tools/latest`.
* `export ANDROID_HOME=/path/to/android-sdk`


## Building

Building for Android requires doing a cross-build where you have a "build"
Python to help produce an Android build of CPython. This procedure has been
tested on Linux and macOS.

The easiest way to do a build is to use the `android.py` script. You can either
have it perform the entire build process from start to finish in one step, or
you can do it in discrete steps that mirror running `configure` and `make` for
each of the two builds of Python you end up producing.

The discrete steps for building via `android.py` are:

```sh
./android.py configure-build
./android.py make-build
./android.py configure-host HOST
./android.py make-host HOST
```

To see the possible values of HOST, run `./android.py configure-host --help`.

Or to do it all in a single command, run:

```sh
./android.py build HOST
```

In the end you should have a build Python in `cross-build/build`, and an Android
build in `cross-build/HOST`.

You can use `--` as a separator for any of the `configure`-related commands –
including `build` itself – to pass arguments to the underlying `configure`
call. For example, if you want a pydebug build that also caches the results from
`configure`, you can do:

```sh
./android.py build HOST -- -C --with-pydebug
```
87 changes: 87 additions & 0 deletions Android/android-env.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
# This script must be sourced with the following variables already set:
: ${ANDROID_HOME:?} # Path to Android SDK
: ${HOST:?} # GNU target triplet

# You may also override the following:
: ${api_level:=21} # Minimum Android API level the build will run on
: ${PREFIX:-} # Path in which to find required libraries


# Print all messages on stderr so they're visible when running within build-wheel.
log() {
echo "$1" >&2
}

fail() {
log "$1"
exit 1
}

# When moving to a new version of the NDK, carefully review the following:
#
# * https://developer.android.com/ndk/downloads/revision_history
#
# * https://android.googlesource.com/platform/ndk/+/ndk-rXX-release/docs/BuildSystemMaintainers.md
# where XX is the NDK version. Do a diff against the version you're upgrading from, e.g.:
# https://android.googlesource.com/platform/ndk/+/ndk-r25-release..ndk-r26-release/docs/BuildSystemMaintainers.md
ndk_version=26.2.11394342

ndk=$ANDROID_HOME/ndk/$ndk_version
if ! [ -e $ndk ]; then
log "Installing NDK: this may take several minutes"
yes | $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager "ndk;$ndk_version"
fi

if [ $HOST = "arm-linux-androideabi" ]; then
clang_triplet=armv7a-linux-androideabi
else
clang_triplet=$HOST
fi

# These variables are based on BuildSystemMaintainers.md above, and
# $ndk/build/cmake/android.toolchain.cmake.
toolchain=$(echo $ndk/toolchains/llvm/prebuilt/*)
export AR="$toolchain/bin/llvm-ar"
export AS="$toolchain/bin/llvm-as"
export CC="$toolchain/bin/${clang_triplet}${api_level}-clang"
export CXX="${CC}++"
export LD="$toolchain/bin/ld"
export NM="$toolchain/bin/llvm-nm"
export RANLIB="$toolchain/bin/llvm-ranlib"
export READELF="$toolchain/bin/llvm-readelf"
export STRIP="$toolchain/bin/llvm-strip"

# The quotes make sure the wildcard in the `toolchain` assignment has been expanded.
for path in "$AR" "$AS" "$CC" "$CXX" "$LD" "$NM" "$RANLIB" "$READELF" "$STRIP"; do
if ! [ -e "$path" ]; then
fail "$path does not exist"
fi
done

export CFLAGS=""
export LDFLAGS="-Wl,--build-id=sha1 -Wl,--no-rosegment"

# Many packages get away with omitting -lm on Linux, but Android is stricter.
LDFLAGS="$LDFLAGS -lm"

# -mstackrealign is included where necessary in the clang launcher scripts which are
# pointed to by $CC, so we don't need to include it here.
if [ $HOST = "arm-linux-androideabi" ]; then
CFLAGS="$CFLAGS -march=armv7-a -mthumb"
fi

if [ -n "${PREFIX:-}" ]; then
abs_prefix=$(realpath $PREFIX)
CFLAGS="$CFLAGS -I$abs_prefix/include"
LDFLAGS="$LDFLAGS -L$abs_prefix/lib"

export PKG_CONFIG="pkg-config --define-prefix"
export PKG_CONFIG_LIBDIR="$abs_prefix/lib/pkgconfig"
fi

# Use the same variable name as conda-build
if [ $(uname) = "Darwin" ]; then
export CPU_COUNT=$(sysctl -n hw.ncpu)
else
export CPU_COUNT=$(nproc)
fi
202 changes: 202 additions & 0 deletions Android/android.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
#!/usr/bin/env python3

import argparse
import os
import re
import shutil
import subprocess
import sys
import sysconfig
from os.path import relpath
from pathlib import Path

SCRIPT_NAME = Path(__file__).name
CHECKOUT = Path(__file__).resolve().parent.parent
CROSS_BUILD_DIR = CHECKOUT / "cross-build"


def delete_if_exists(path):
if path.exists():
print(f"Deleting {path} ...")
shutil.rmtree(path)


def subdir(name, *, clean=None):
path = CROSS_BUILD_DIR / name
if clean:
delete_if_exists(path)
if not path.exists():
if clean is None:
sys.exit(
f"{path} does not exist. Create it by running the appropriate "
f"`configure` subcommand of {SCRIPT_NAME}.")
else:
path.mkdir(parents=True)
return path


def run(command, *, host=None, **kwargs):
env = os.environ.copy()
if host:
env_script = CHECKOUT / "Android/android-env.sh"
env_output = subprocess.run(
f"set -eu; "
f"HOST={host}; "
f"PREFIX={subdir(host)}/prefix; "
f". {env_script}; "
f"export",
check=True, shell=True, text=True, stdout=subprocess.PIPE
).stdout

for line in env_output.splitlines():
# We don't require every line to match, as there may be some other
# output from installing the NDK.
if match := re.search(
"^(declare -x |export )?(\\w+)=['\"]?(.*?)['\"]?$", line
):
key, value = match[2], match[3]
if env.get(key) != value:
print(line)
env[key] = value

if env == os.environ:
raise ValueError(f"Found no variables in {env_script.name} output:\n"
+ env_output)

print(">", " ".join(map(str, command)))
try:
subprocess.run(command, check=True, env=env, **kwargs)
except subprocess.CalledProcessError as e:
sys.exit(e)


def build_python_path():
"""The path to the build Python binary."""
build_dir = subdir("build")
binary = build_dir / "python"
if not binary.is_file():
binary = binary.with_suffix(".exe")
if not binary.is_file():
raise FileNotFoundError("Unable to find `python(.exe)` in "
f"{build_dir}")

return binary


def configure_build_python(context):
os.chdir(subdir("build", clean=context.clean))

command = [relpath(CHECKOUT / "configure")]
if context.args:
command.extend(context.args)
run(command)


def make_build_python(context):
os.chdir(subdir("build"))
run(["make", "-j", str(os.cpu_count())])


def unpack_deps(host):
deps_url = "https://github.com/beeware/cpython-android-source-deps/releases/download"
for name_ver in ["bzip2-1.0.8-1", "libffi-3.4.4-2", "openssl-3.0.13-1",
"sqlite-3.45.1-0", "xz-5.4.6-0"]:
filename = f"{name_ver}-{host}.tar.gz"
run(["wget", f"{deps_url}/{name_ver}/{filename}"])
run(["tar", "-xf", filename])
os.remove(filename)


def configure_host_python(context):
host_dir = subdir(context.host, clean=context.clean)

prefix_dir = host_dir / "prefix"
if not prefix_dir.exists():
prefix_dir.mkdir()
os.chdir(prefix_dir)
unpack_deps(context.host)

build_dir = host_dir / "build"
build_dir.mkdir(exist_ok=True)
os.chdir(build_dir)

command = [
# Basic cross-compiling configuration
relpath(CHECKOUT / "configure"),
f"--host={context.host}",
f"--build={sysconfig.get_config_var('BUILD_GNU_TYPE')}",
f"--with-build-python={build_python_path()}",
"--without-ensurepip",

# Android always uses a shared libpython.
"--enable-shared",
"--without-static-libpython",

# Dependent libraries. The others are found using pkg-config: see
# android-env.sh.
f"--with-openssl={prefix_dir}",
]

if context.args:
command.extend(context.args)
run(command, host=context.host)


def make_host_python(context):
host_dir = subdir(context.host)
os.chdir(host_dir / "build")
run(["make", "-j", str(os.cpu_count())], host=context.host)
run(["make", "install", f"prefix={host_dir}/prefix"], host=context.host)


def build_all(context):
steps = [configure_build_python, make_build_python, configure_host_python,
make_host_python]
for step in steps:
step(context)


def clean_all(context):
delete_if_exists(CROSS_BUILD_DIR)


def main():
parser = argparse.ArgumentParser()
subcommands = parser.add_subparsers(dest="subcommand")
build = subcommands.add_parser("build", help="Build everything")
configure_build = subcommands.add_parser("configure-build",
help="Run `configure` for the "
"build Python")
make_build = subcommands.add_parser("make-build",
help="Run `make` for the build Python")
configure_host = subcommands.add_parser("configure-host",
help="Run `configure` for Android")
make_host = subcommands.add_parser("make-host",
help="Run `make` for Android")
clean = subcommands.add_parser("clean", help="Delete files and directories "
"created by this script")
for subcommand in build, configure_build, configure_host:
subcommand.add_argument(
"--clean", action="store_true", default=False, dest="clean",
help="Delete any relevant directories before building")
for subcommand in build, configure_host, make_host:
subcommand.add_argument(
"host", metavar="HOST",
choices=["aarch64-linux-android", "x86_64-linux-android"],
help="Host triplet: choices=[%(choices)s]")
for subcommand in build, configure_build, configure_host:
subcommand.add_argument("args", nargs="*",
help="Extra arguments to pass to `configure`")

context = parser.parse_args()
dispatch = {"configure-build": configure_build_python,
"make-build": make_build_python,
"configure-host": configure_host_python,
"make-host": make_host_python,
"build": build_all,
"clean": clean_all}
dispatch[context.subcommand](context)


if __name__ == "__main__":
main()
1 change: 1 addition & 0 deletions Doc/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ venv:
echo "venv already exists."; \
echo "To recreate it, remove it first with \`make clean-venv'."; \
else \
echo "Creating venv in $(VENVDIR)"; \
$(PYTHON) -m venv $(VENVDIR); \
$(VENVDIR)/bin/python3 -m pip install --upgrade pip; \
$(VENVDIR)/bin/python3 -m pip install -r $(REQUIREMENTS); \
Expand Down
Loading

0 comments on commit a26c4b0

Please sign in to comment.