Skip to content

Commit

Permalink
pythongh-126925: Modify how iOS test results are gathered (python#127592
Browse files Browse the repository at this point in the history
)

Adds a `use_system_log` config item to enable stdout/stderr redirection for
Apple platforms. This log streaming is then used by a new iOS test runner
script, allowing the display of test suite output at runtime. The iOS test
runner script can be used by any Python project, not just the CPython test
suite.
  • Loading branch information
freakboy3742 committed Dec 13, 2024
1 parent e750f21 commit b2be118
Show file tree
Hide file tree
Showing 18 changed files with 792 additions and 58 deletions.
9 changes: 9 additions & 0 deletions Doc/c-api/init_config.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1279,6 +1279,15 @@ PyConfig
Default: ``1`` in Python config and ``0`` in isolated config.
.. c:member:: int use_system_logger
If non-zero, ``stdout`` and ``stderr`` will be redirected to the system
log.
Only available on macOS 10.12 and later, and on iOS.
Default: ``0`` (don't use system log).
.. c:member:: int user_site_directory
If non-zero, add the user site directory to :data:`sys.path`.
Expand Down
53 changes: 49 additions & 4 deletions Doc/using/ios.rst
Original file line number Diff line number Diff line change
Expand Up @@ -292,10 +292,12 @@ To add Python to an iOS Xcode project:
10. Add Objective C code to initialize and use a Python interpreter in embedded
mode. You should ensure that:

* :c:member:`UTF-8 mode <PyPreConfig.utf8_mode>` is *enabled*;
* :c:member:`Buffered stdio <PyConfig.buffered_stdio>` is *disabled*;
* :c:member:`Writing bytecode <PyConfig.write_bytecode>` is *disabled*;
* :c:member:`Signal handlers <PyConfig.install_signal_handlers>` are *enabled*;
* UTF-8 mode (:c:member:`PyPreConfig.utf8_mode`) is *enabled*;
* Buffered stdio (:c:member:`PyConfig.buffered_stdio`) is *disabled*;
* Writing bytecode (:c:member:`PyConfig.write_bytecode`) is *disabled*;
* Signal handlers (:c:member:`PyConfig.install_signal_handlers`) are *enabled*;
* System logging (:c:member:`PyConfig.use_system_logger`) is *enabled*
(optional, but strongly recommended);
* ``PYTHONHOME`` for the interpreter is configured to point at the
``python`` subfolder of your app's bundle; and
* The ``PYTHONPATH`` for the interpreter includes:
Expand Down Expand Up @@ -324,6 +326,49 @@ modules in your app, some additional steps will be required:
* If you're using a separate folder for third-party packages, ensure that folder
is included as part of the ``PYTHONPATH`` configuration in step 10.

Testing a Python package
------------------------

The CPython source tree contains :source:`a testbed project <iOS/testbed>` that
is used to run the CPython test suite on the iOS simulator. This testbed can also
be used as a testbed project for running your Python library's test suite on iOS.

After building or obtaining an iOS XCFramework (See :source:`iOS/README.rst`
for details), create a clone of the Python iOS testbed project by running:

.. code-block:: bash
$ python iOS/testbed clone --framework <path/to/Python.xcframework> --app <path/to/module1> --app <path/to/module2> app-testbed
You will need to modify the ``iOS/testbed`` reference to point to that
directory in the CPython source tree; any folders specified with the ``--app``
flag will be copied into the cloned testbed project. The resulting testbed will
be created in the ``app-testbed`` folder. In this example, the ``module1`` and
``module2`` would be importable modules at runtime. If your project has
additional dependencies, they can be installed into the
``app-testbed/iOSTestbed/app_packages`` folder (using ``pip install --target
app-testbed/iOSTestbed/app_packages`` or similar).

You can then use the ``app-testbed`` folder to run the test suite for your app,
For example, if ``module1.tests`` was the entry point to your test suite, you
could run:

.. code-block:: bash
$ python app-testbed run -- module1.tests
This is the equivalent of running ``python -m module1.tests`` on a desktop
Python build. Any arguments after the ``--`` will be passed to the testbed as
if they were arguments to ``python -m`` on a desktop machine.

You can also open the testbed project in Xcode by running:

.. code-block:: bash
$ open app-testbed/iOSTestbed.xcodeproj
This will allow you to use the full Xcode suite of tools for debugging.

App Store Compliance
====================

Expand Down
7 changes: 7 additions & 0 deletions Doc/whatsnew/3.14.rst
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,13 @@ Other language changes
making it a :term:`generic type`.
(Contributed by Brian Schubert in :gh:`126012`.)

* iOS and macOS apps can now be configured to redirect ``stdout`` and
``stderr`` content to the system log. (Contributed by Russell Keith-Magee in
:gh:`127592`.)

* The iOS testbed is now able to stream test output while the test is running.
The testbed can also be used to run the test suite of projects other than
CPython itself. (Contributed by Russell Keith-Magee in :gh:`127592`.)

New modules
===========
Expand Down
3 changes: 3 additions & 0 deletions Include/cpython/initconfig.h
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,9 @@ typedef struct PyConfig {
int use_frozen_modules;
int safe_path;
int int_max_str_digits;
#ifdef __APPLE__
int use_system_logger;
#endif

int cpu_count;
#ifdef Py_GIL_DISABLED
Expand Down
66 changes: 66 additions & 0 deletions Lib/_apple_support.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
import io
import sys


def init_streams(log_write, stdout_level, stderr_level):
# Redirect stdout and stderr to the Apple system log. This method is
# invoked by init_apple_streams() (initconfig.c) if config->use_system_logger
# is enabled.
sys.stdout = SystemLog(log_write, stdout_level, errors=sys.stderr.errors)
sys.stderr = SystemLog(log_write, stderr_level, errors=sys.stderr.errors)


class SystemLog(io.TextIOWrapper):
def __init__(self, log_write, level, **kwargs):
kwargs.setdefault("encoding", "UTF-8")
kwargs.setdefault("line_buffering", True)
super().__init__(LogStream(log_write, level), **kwargs)

def __repr__(self):
return f"<SystemLog (level {self.buffer.level})>"

def write(self, s):
if not isinstance(s, str):
raise TypeError(
f"write() argument must be str, not {type(s).__name__}")

# In case `s` is a str subclass that writes itself to stdout or stderr
# when we call its methods, convert it to an actual str.
s = str.__str__(s)

# We want to emit one log message per line, so split
# the string before sending it to the superclass.
for line in s.splitlines(keepends=True):
super().write(line)

return len(s)


class LogStream(io.RawIOBase):
def __init__(self, log_write, level):
self.log_write = log_write
self.level = level

def __repr__(self):
return f"<LogStream (level {self.level!r})>"

def writable(self):
return True

def write(self, b):
if type(b) is not bytes:
try:
b = bytes(memoryview(b))
except TypeError:
raise TypeError(
f"write() argument must be bytes-like, not {type(b).__name__}"
) from None

# Writing an empty string to the stream should have no effect.
if b:
# Encode null bytes using "modified UTF-8" to avoid truncating the
# message. This should not affect the return value, as the caller
# may be expecting it to match the length of the input.
self.log_write(self.level, b.replace(b"\x00", b"\xc0\x80"))

return len(b)
155 changes: 155 additions & 0 deletions Lib/test/test_apple.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
import unittest
from _apple_support import SystemLog
from test.support import is_apple
from unittest.mock import Mock, call

if not is_apple:
raise unittest.SkipTest("Apple-specific")


# Test redirection of stdout and stderr to the Apple system log.
class TestAppleSystemLogOutput(unittest.TestCase):
maxDiff = None

def assert_writes(self, output):
self.assertEqual(
self.log_write.mock_calls,
[
call(self.log_level, line)
for line in output
]
)

self.log_write.reset_mock()

def setUp(self):
self.log_write = Mock()
self.log_level = 42
self.log = SystemLog(self.log_write, self.log_level, errors="replace")

def test_repr(self):
self.assertEqual(repr(self.log), "<SystemLog (level 42)>")
self.assertEqual(repr(self.log.buffer), "<LogStream (level 42)>")

def test_log_config(self):
self.assertIs(self.log.writable(), True)
self.assertIs(self.log.readable(), False)

self.assertEqual("UTF-8", self.log.encoding)
self.assertEqual("replace", self.log.errors)

self.assertIs(self.log.line_buffering, True)
self.assertIs(self.log.write_through, False)

def test_empty_str(self):
self.log.write("")
self.log.flush()

self.assert_writes([])

def test_simple_str(self):
self.log.write("hello world\n")

self.assert_writes([b"hello world\n"])

def test_buffered_str(self):
self.log.write("h")
self.log.write("ello")
self.log.write(" ")
self.log.write("world\n")
self.log.write("goodbye.")
self.log.flush()

self.assert_writes([b"hello world\n", b"goodbye."])

def test_manual_flush(self):
self.log.write("Hello")

self.assert_writes([])

self.log.write(" world\nHere for a while...\nGoodbye")
self.assert_writes([b"Hello world\n", b"Here for a while...\n"])

self.log.write(" world\nHello again")
self.assert_writes([b"Goodbye world\n"])

self.log.flush()
self.assert_writes([b"Hello again"])

def test_non_ascii(self):
# Spanish
self.log.write("ol\u00e9\n")
self.assert_writes([b"ol\xc3\xa9\n"])

# Chinese
self.log.write("\u4e2d\u6587\n")
self.assert_writes([b"\xe4\xb8\xad\xe6\x96\x87\n"])

# Printing Non-BMP emoji
self.log.write("\U0001f600\n")
self.assert_writes([b"\xf0\x9f\x98\x80\n"])

# Non-encodable surrogates are replaced
self.log.write("\ud800\udc00\n")
self.assert_writes([b"??\n"])

def test_modified_null(self):
# Null characters are logged using "modified UTF-8".
self.log.write("\u0000\n")
self.assert_writes([b"\xc0\x80\n"])
self.log.write("a\u0000\n")
self.assert_writes([b"a\xc0\x80\n"])
self.log.write("\u0000b\n")
self.assert_writes([b"\xc0\x80b\n"])
self.log.write("a\u0000b\n")
self.assert_writes([b"a\xc0\x80b\n"])

def test_nonstandard_str(self):
# String subclasses are accepted, but they should be converted
# to a standard str without calling any of their methods.
class CustomStr(str):
def splitlines(self, *args, **kwargs):
raise AssertionError()

def __len__(self):
raise AssertionError()

def __str__(self):
raise AssertionError()

self.log.write(CustomStr("custom\n"))
self.assert_writes([b"custom\n"])

def test_non_str(self):
# Non-string classes are not accepted.
for obj in [b"", b"hello", None, 42]:
with self.subTest(obj=obj):
with self.assertRaisesRegex(
TypeError,
fr"write\(\) argument must be str, not "
fr"{type(obj).__name__}"
):
self.log.write(obj)

def test_byteslike_in_buffer(self):
# The underlying buffer *can* accept bytes-like objects
self.log.buffer.write(bytearray(b"hello"))
self.log.flush()

self.log.buffer.write(b"")
self.log.flush()

self.log.buffer.write(b"goodbye")
self.log.flush()

self.assert_writes([b"hello", b"goodbye"])

def test_non_byteslike_in_buffer(self):
for obj in ["hello", None, 42]:
with self.subTest(obj=obj):
with self.assertRaisesRegex(
TypeError,
fr"write\(\) argument must be bytes-like, not "
fr"{type(obj).__name__}"
):
self.log.buffer.write(obj)
4 changes: 4 additions & 0 deletions Lib/test/test_capi/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,10 @@ def test_config_get(self):
options.extend((
("_pystats", bool, None),
))
if support.is_apple:
options.extend((
("use_system_logger", bool, None),
))

for name, option_type, sys_attr in options:
with self.subTest(name=name, option_type=option_type,
Expand Down
2 changes: 2 additions & 0 deletions Lib/test/test_embed.py
Original file line number Diff line number Diff line change
Expand Up @@ -649,6 +649,8 @@ class InitConfigTests(EmbeddingTestsMixin, unittest.TestCase):
CONFIG_COMPAT.update({
'legacy_windows_stdio': False,
})
if support.is_apple:
CONFIG_COMPAT['use_system_logger'] = False

CONFIG_PYTHON = dict(CONFIG_COMPAT,
_config_init=API_PYTHON,
Expand Down
26 changes: 4 additions & 22 deletions Makefile.pre.in
Original file line number Diff line number Diff line change
Expand Up @@ -2132,7 +2132,6 @@ testuniversal: all
# This must be run *after* a `make install` has completed the build. The
# `--with-framework-name` argument *cannot* be used when configuring the build.
XCFOLDER:=iOSTestbed.$(MULTIARCH).$(shell date +%s)
XCRESULT=$(XCFOLDER)/$(MULTIARCH).xcresult
.PHONY: testios
testios:
@if test "$(MACHDEP)" != "ios"; then \
Expand All @@ -2151,29 +2150,12 @@ testios:
echo "Cannot find a finalized iOS Python.framework. Have you run 'make install' to finalize the framework build?"; \
exit 1;\
fi
# Copy the testbed project into the build folder
cp -r $(srcdir)/iOS/testbed $(XCFOLDER)
# Copy the framework from the install location to the testbed project.
cp -r $(PYTHONFRAMEWORKPREFIX)/* $(XCFOLDER)/Python.xcframework/ios-arm64_x86_64-simulator

# Run the test suite for the Xcode project, targeting the iOS simulator.
# If the suite fails, touch a file in the test folder as a marker
if ! xcodebuild test -project $(XCFOLDER)/iOSTestbed.xcodeproj -scheme "iOSTestbed" -destination "platform=iOS Simulator,name=iPhone SE (3rd Generation)" -resultBundlePath $(XCRESULT) -derivedDataPath $(XCFOLDER)/DerivedData ; then \
touch $(XCFOLDER)/failed; \
fi

# Regardless of success or failure, extract and print the test output
xcrun xcresulttool get --path $(XCRESULT) \
--id $$( \
xcrun xcresulttool get --path $(XCRESULT) --format json | \
$(PYTHON_FOR_BUILD) -c "import sys, json; result = json.load(sys.stdin); print(result['actions']['_values'][0]['actionResult']['logRef']['id']['_value'])" \
) \
--format json | \
$(PYTHON_FOR_BUILD) -c "import sys, json; result = json.load(sys.stdin); print(result['subsections']['_values'][1]['subsections']['_values'][0]['emittedOutput']['_value'])"
# Clone the testbed project into the XCFOLDER
$(PYTHON_FOR_BUILD) $(srcdir)/iOS/testbed clone --framework $(PYTHONFRAMEWORKPREFIX) "$(XCFOLDER)"

@if test -e $(XCFOLDER)/failed ; then \
exit 1; \
fi
# Run the testbed project
$(PYTHON_FOR_BUILD) "$(XCFOLDER)" run -- test -uall --single-process --rerun -W

# Like test, but using --slow-ci which enables all test resources and use
# longer timeout. Run an optional pybuildbot.identify script to include
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
macOS and iOS apps can now choose to redirect stdout and stderr to the
system log during interpreter configuration.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
iOS test results are now streamed during test execution, and the deprecated
xcresulttool is no longer used.
Loading

0 comments on commit b2be118

Please sign in to comment.