Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-44421: [Python] Add configuration for building & testing free-threaded wheels on Windows #44804

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

lysnikolaou
Copy link
Contributor

@lysnikolaou lysnikolaou commented Nov 21, 2024

Rationale for this change

There's no blockers anymore for building Windows wheels for the free-threaded build, so this PR adds the necessary configuration to do that.

What changes are included in this PR?

  • Add jobs to build free-threaded wheels on Windows
  • Move VCPKG-related stuff to a reusable base job

Are these changes tested?

No tests required.

Are there any user-facing changes?

No.

Copy link

⚠️ GitHub issue #44421 has been automatically assigned in GitHub to PR creator.

@github-actions github-actions bot added the awaiting review Awaiting review label Nov 21, 2024
@lysnikolaou
Copy link
Contributor Author

If someone could trigger the new jobs, that'd be really helpful!

@raulcd
Copy link
Member

raulcd commented Nov 27, 2024

@github-actions crossbow submit -g wheel-windows-*

Copy link

Invalid group(s) {'wheel-windows-*'}. Must be one of {'wheel', 'cuda', 'example-cpp', 'linux', 'homebrew', 'fuzz', 'integration', 'cpp', 'example-python', 'nightly-release', 'verify-rc-source-linux', 'c-glib', 'example', 'python', 'nightly', 'conan', 'verify-rc-wheels', 'packaging', 'r', 'verify-rc-source', 'verify-rc', 'nightly-packaging', 'nightly-tests', 'vcpkg', 'java', 'linux-arm64', 'verify-rc-binaries', 'conda', 'test', 'verify-rc-source-macos', 'ruby', 'verify-rc-jars', 'linux-amd64'}
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/12047528593

@raulcd
Copy link
Member

raulcd commented Nov 27, 2024

@github-actions crossbow submit wheel-windows-*

Copy link

Revision: 78d8f9e

Submitted crossbow builds: ursacomputing/crossbow @ actions-86ba33e477

Task Status
wheel-windows-cp310-cp310-amd64 GitHub Actions
wheel-windows-cp311-cp311-amd64 GitHub Actions
wheel-windows-cp312-cp312-amd64 GitHub Actions
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions
wheel-windows-cp39-cp39-amd64 GitHub Actions

Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, I triggered the CI jobs

@github-actions github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting review Awaiting review awaiting changes Awaiting changes labels Nov 27, 2024
@lysnikolaou
Copy link
Contributor Author

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

@lysnikolaou
Copy link
Contributor Author

Also, the failure on 3.9 is that py cannot be found when running py -%PYTHON% setup.py bdist_wheel || exit /B 1, which seems odd.

@raulcd
Copy link
Member

raulcd commented Dec 2, 2024

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

I haven't tested and I am unsure if this works but maybe we can try this (https://stackoverflow.com/a/21912169):
py -3.13t -c "import pyarrow" || echo ERROR && exit /B 1 instead of py -3.13t -c "import pyarrow" || exit /B 1

@lysnikolaou
Copy link
Contributor Author

The Windows failure is probably related to pypa/setuptools#4662 and should be fixed in the next setuptools release.

@pitrou
Copy link
Member

pitrou commented Dec 4, 2024

The Windows failure is probably related to pypa/setuptools#4662 and should be fixed in the next setuptools release.

Hmm, that one should be fixed in CPython AFAIU: python/cpython#111650

@pitrou
Copy link
Member

pitrou commented Dec 4, 2024

Ah, scratch that, the rationale is actually here:
https://github.com/pypa/distutils/pull/310/files

@pitrou
Copy link
Member

pitrou commented Dec 4, 2024

Unfortunately, while the free-threaded wheel is built correctly, py -3.13t -c "import pyarrow" fails. No error shown. Does anyone know how I could inspect stderr to see why this fails?

Well, stderr should already be shown by default, so it's weird that nothing gets displayed.

@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from 95a0a53 to 83a76e3 Compare December 18, 2024 17:31
@pitrou
Copy link
Member

pitrou commented Dec 18, 2024

@github-actions crossbow submit wheel-windows-*

@pitrou
Copy link
Member

pitrou commented Dec 18, 2024

Given setuptools does not seem in a hurry to merge the distutils fix, I've tried to push a manual workaround in this PR.

Copy link

Revision: 83a76e3

Submitted crossbow builds: ursacomputing/crossbow @ actions-5bfc22ac30

Task Status
wheel-windows-cp310-cp310-amd64 GitHub Actions
wheel-windows-cp311-cp311-amd64 GitHub Actions
wheel-windows-cp312-cp312-amd64 GitHub Actions
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions
wheel-windows-cp39-cp39-amd64 GitHub Actions

@pitrou
Copy link
Member

pitrou commented Dec 18, 2024

Well, building the free-threaded wheel failed with:

LINK : fatal error LNK1104: cannot open file 'python313t.lib' [C:\arrow\python\build\temp.win-amd64-cpython-313t\arrow_python.vcxproj]

Edit: I've checked: that file exists in a Python 3.13 Windows install.

@pitrou
Copy link
Member

pitrou commented Dec 18, 2024

@github-actions crossbow submit wheel-windows-cp313*

@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from 0d30b99 to 55e4cc0 Compare December 19, 2024 07:42
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

Copy link

Revision: 55e4cc0

Submitted crossbow builds: ursacomputing/crossbow @ actions-1076ab812c

Task Status
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions

@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

This comment was marked as outdated.

@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from e4c9baf to 184510c Compare December 19, 2024 10:20
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

This comment was marked as outdated.

@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from 184510c to b5fbae1 Compare December 19, 2024 10:51
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

This comment was marked as outdated.

@pitrou pitrou force-pushed the windows-free-threaded-wheels branch from b5fbae1 to 771cf66 Compare December 19, 2024 13:10
@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

This comment was marked as outdated.

@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

This comment was marked as outdated.

@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

Copy link

Revision: da27ca9

Submitted crossbow builds: ursacomputing/crossbow @ actions-4bc1383e5a

Task Status
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions

@pitrou
Copy link
Member

pitrou commented Dec 19, 2024

@github-actions crossbow submit wheel-windows-cp313*

Copy link

Revision: de2f14a

Submitted crossbow builds: ursacomputing/crossbow @ actions-bfacfef460

Task Status
wheel-windows-cp313-cp313-amd64 GitHub Actions
wheel-windows-cp313-cp313t-amd64 GitHub Actions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants