gh-126925: Modify how iOS test results are gathered #127592
Conversation
freakboy3742
added
tests
freakboy3742
requested review from
FFY00,
ericsnowcurrently and
erlend-aasland
as code owners
|
!buildbot iOS |
|
🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit 60584e0 🤖 The command will test the builders whose names match following regular expression: The builders matched are:
|
|
Moving to draft because of the test failures on macOS buils |
|
!buildbot iOS |
|
🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit 89bb435 🤖 The command will test the builders whose names match following regular expression: The builders matched are:
|
|
I've now resolved the testing issue; I've also added some documentation for the new capabilities. |
Agreed - this is a case where it's the iOS logger being messy, and this script is just returning the mess verbatim. However, it's something we might be able to clean up by checking for that specific content, and swallowing all but the first occurrence.
Ack - I'll add both these two. |
|
!buildbot iOS |
|
🤖 New build scheduled with the buildbot fleet by @freakboy3742 for commit 47870c4 🤖 The command will test the builders whose names match following regular expression: The builders matched are:
|
There's no command line interface; it's currently only exposed as an option for embedded-mode apps. Briefcase's stub binary, for example, would be able to add stdout redirection as part of the app startup sequence, rather than requiring the @ned-deily I've now made those changes, including surpassing all but the first occurrence of the "Messages dropped..." messages in any block when they occur. Did you want to review this again, or are you OK with me to merge and backport the changes? |
|
LGTM, thanks |
|
Thanks @freakboy3742 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13. |
|
Sorry, @freakboy3742, I could not cleanly backport this to |
…honGH-127592) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95) Co-authored-by: Russell Keith-Magee <[email protected]>
|
GH-127754 is a backport of this pull request to the 3.13 branch. |
#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
…onGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite.
…honGH-127592) (python#127754) Adds a `use_system_log` config item to enable stdout/stderr redirection for Apple platforms. This log streaming is then used by a new iOS test runner script, allowing the display of test suite output at runtime. The iOS test runner script can be used by any Python project, not just the CPython test suite. (cherry picked from commit 2041a95)
Xcode 16 deprecated the
xcresultooltool we were using to gather test results after an iOS test run.This tool was less than ideal anyway, as it didn't provide a way to see test results as the test suite was running. As a result, we had intermittent CI test failures that manifested as "test suite didn't finish", with no runtime debugging possible.
This PR adds a new configuration item (
config->use_system_logger) for Apple platforms, providing the option to redirect stdout and stderr to the system log. This mirrors how Android operates - the implementation is strongly influenced by the Android implementation, but is just different enough that refactoring into a common base class is more trouble than it's worth IMHO.This PR also adds a new test runner for iOS projects. This test runner uses the iOS testbed as a template, installing the compiled framework into copy generated from the template; and then building and running the test project in parallel with a log streamer that captures the system log from the iOS simulator as it runs. This allows us to observe test output as the test suite runs.
The test runner also includes options to allows the installation of app code. This means the testbed project can be used to test arbitrary Python code, in addition to the core CPython test suite.
Although this feature is primarily useful for iOS apps, the same code can be used for macOS apps - something that is useful when embedding Python in to a GUI app (where stdout/stderr isn't visible).
This PR resolves both #126925 and #126821; Resolving #126821 was a pre-requisite for fixing #126925 (and is the first commit in this PR), but by itself, there's very little opportunity to evaluate that it's working. You need an updated test runner to be able to observe that stdout/err redirection is working.
Fixes #126925
Fixes #126821
📚 Documentation preview 📚: https://cpython-previews--127592.org.readthedocs.build/