feat: support pypa build #521

henryiii · 2021-01-06T15:22:21Z

This will close #442.

henryiii · 2021-01-06T15:36:20Z

Might be easier if pypa/manylinux#941 is accepted.

henryiii · 2021-01-06T17:02:47Z

pypa-build builds in a virtual environment, so the path check in setup.py test is incorrect. It sees the executable we require in the first step, and the build virtual environment path in the second. Any ideas about how to change the check? Wouldn't an exact version check be enough, perhaps? We could provide a second version of it without this check? Etc.

henryiii · 2021-01-06T22:33:22Z

This should probably be seen as a proof of concept until pypa/manylinux#941 goes in, because we currently don't touch the contents of the manylinux image.

joerick · 2021-01-15T13:57:27Z

BTW, I wouldn't be opposed to releasing PyPA build support before pypa/manylinux#941, given that this would be advertised as 'experimental' for now anyway.

henryiii · 2021-03-25T13:07:46Z

I've included pypa/build in manylinux now, including manylinux1 for Python 2.7 (!), but we can't use it because we've pinned manylinux2010 to a version before that addition.

Honestly, I don't think dropping 2.7 support for manylinux2010 but keeping it for manylinux1 for now would be that big of an issue for a 1.11 release. We have very few users using manylinux2010 anyway, and even fewer of them making 2.7 wheels. Requires-Python support already helps, and we could check to see what Pythons are in the image and limit to that, producing an error if any are missed that are included by Requires-Python, and a warning if Requires-Python is not set.

henryiii · 2021-05-07T21:16:37Z

Does the PyPy image not contain build?

mayeut · 2021-05-08T08:41:55Z

It seems they are base on an old manylinux2010 image so no pypa/build yet.
pypy/manylinux#16 should resolve this once merged.

mayeut · 2021-05-08T16:15:37Z

pypy/manylinux#16 has been merged. The new image should have build (& auditwheel 4)

henryiii · 2021-05-17T22:07:05Z

Actually, it looks like pypy/manylinux is based on an old version of pypa/manylinux; notably it has a copy of requirements.txt internally (which is missing other important things we've added in this repo, including build).

Originally posted by @henryiii in pypa/manylinux#1055 (comment)

This means no build in pypy/manylinux, which is why this PR is failing.

mayeut · 2021-05-24T13:58:23Z

Switched to quay.io/pypa/manylinux2010_x86_64 for PyPy while waiting for #671 just to see what the tests look like on GHA (need to rebase to get CI green)

henryiii · 2021-05-25T04:45:50Z

Windows is failing on PyPy 7.3.3 + build:

Unable to copy 'C:\\cibw\\pypy3.7-v7.3.3-win32\\venvlauncher.exe'

mayeut · 2021-05-25T17:29:54Z

Same error with PyPy 7.3.5

henryiii · 2021-05-27T04:18:21Z

pypa/build is tested on Windows + PyPy{2,3}, so not sure where this error is coming from.

I also think there are a few tests I need to update the output checking on, will look into that later.

mayeut · 2021-05-27T12:16:14Z

@henryiii, rebased on main to get CI again. I fixed the cibuildwheel.linux.troubleshoot function to get all builds passing locally on linux. Let's see if CI agrees, there will only be the Windows PyPy issue left to investigate I think.

henryiii · 2021-05-27T20:20:39Z

Looks like it's checking the exact pip versions, which is not true in the env created by build.

mayeut · 2021-05-30T06:49:43Z

Looks like it's checking the exact pip versions, which is not true in the env created by build.

This is the issue for macOS.
For Windows PyPy, something else is going on.

mayeut · 2021-05-31T09:36:19Z

I investigated a bit the issue with PyPy on Windows.
It seems the venv module does not like to be called from the python.exe symlink at all: https://foss.heptapod.net/pypy/pypy/-/issues/3479
Worked around for now by installing build[virtualenv]

mayeut · 2021-05-31T09:57:21Z

The only test failing now is test/test_dependency_versions.py::test_dependency_constraints_file on macOS & Windows i.e.

Looks like it's checking the exact pip versions, which is not true in the env created by build.

~~Should we create/discuss something in pypa/build or should we just ignore this one ?~~ see also pypa/build#292
My 2 cents, it's good to be able to pin things down for reproducibility. We might also want to extend that to linux builds (although it's a different issue) ?

mayeut · 2021-05-31T13:50:47Z

I implemented the hack mentioned in pypa/build#292 just to see if everything passes with it. Not sure it's a good idea to implement such a hack.

henryiii · 2021-06-01T16:51:03Z

cibuildwheel/util.py

+    # we use shell=True here for windows, even though we don't need a shell due to a bug
+    # https://bugs.python.org/issue8557


This would be simpler with #672?

henryiii · 2021-06-03T00:27:52Z

What's the status of this, @mayeut?

mayeut · 2021-06-03T07:08:56Z

Short status:

all tests are passing whether build is the default or not (2e52bf2 really restored pip as the default)
We need to install build[virtualenv] to at least workaround PyPy venv issue, eventually to allow to pin pip when using build.

Open questions / things to do:

Should dependency constraints be kept ? it feels very hacky 6f78d73 / Allow setting build env constraints build#292
Extend tests ? all or select some that seem more relevant ? (already added those showing issues that needed a commit)
Update docs

I'll most likely be offline for the next 2 weeks so not planning on adding more things here.

joerick · 2021-06-03T14:23:10Z

A thought on options design - rather than CIBW_PYPA_BUILD, could we change this to CIBW_BUILD_FRONTEND? (I don't love the name 'frontend' because what does that make us...) but that's the terminology used in PEP 517. Options would be pip or build.

More of a future-proofing measure, really. I've noticed some other tools offering themselves as build frontends... e.g. poetry might start building platform wheels, or something else like https://github.com/FFY00/trampolim might crop up... I suppose that in theory they should 'just' be invoking the build backend, but we can't predict the future!

henryiii · 2021-06-17T21:25:14Z

I'll change CIBW_PYPA_BUILD to CIBW_BUILD_FRONTEND/tool.cibuildwheel.build-frontend after #684.

fix: update `cibuildwheel.linux.troubleshoot` to know about build Workaround issue with PyPy venv module by installing build[virtualenv] fix: test_dependency_constraints_file when using build fix: test/test_pep518.py fix: use `strtobool` to parse `CIBW_PYPA_BUILD` fix: update `cibuildwheel.linux.troubleshoot` to know about `python -m pip wheel` test: use `build_mode` in `test/test_dependency_versions.py` tests

henryiii · 2021-06-21T19:25:40Z

I squashed the previous commits to make it easier to rebase and modify for the new config setup. I then switched to using build-backend. Docs still need to be added.

henryiii · 2021-06-22T03:10:27Z

Put some basic docs in, happy to take improvements.

joerick

Nice one @henryiii and @mayeut ! Just a few tweaks and this is ready to go, I think.

cibuildwheel/__main__.py

cibuildwheel/linux.py

cibuildwheel/macos.py

cibuildwheel/windows.py

cibuildwheel/macos.py

docs/options.md

test/conftest.py

Co-authored-by: Joe Rickerby <[email protected]>

joerick · 2021-06-22T17:05:28Z

oops, i missed these earlier...

Should dependency constraints be kept ? it feels very hacky 6f78d73 / pypa/build#292

Yes, this does seem like a bit of a hack.. the use of PIP_CONSTRAINTS seems like exploiting an implementation detail of build. That said, it's unlikely to be a problem on the user's end, because we pin the version of build. I suppose the other option would be to use --no-isolation and ensure the correct versions of the tools we care about are installed in the environment before invoking build. I'd be okay with either, I think.

Extend tests ? all or select some that seem more relevant ? (already added those showing issues that needed a commit)

Since use of build is only an option, currently, this seems okay for now. I'd be curious if the test suite does pass with CIBW_BUILD_FRONTEND=build, though, did you try with the default as that?

test/test_before_build.py

henryiii · 2021-06-23T14:47:04Z

I suppose the other option would be to use --no-isolation

I think this would break PEP 518; I think the current constraints is preferable for now.

I'd be curious if the test suite does pass with CIBW_BUILD_FRONTEND=build, though, did you try with the default as that?

I think that's how I started, and that's why I had that assert commented out; I was trying to run everything with build. But I could be wrong.

henryiii marked this pull request as draft January 6, 2021 15:25

henryiii force-pushed the feat/pypa-build branch from fb912fd to 1189888 Compare January 6, 2021 15:59

henryiii force-pushed the feat/pypa-build branch from 1189888 to ad0bfff Compare January 6, 2021 23:27

henryiii force-pushed the feat/pypa-build branch 2 times, most recently from 64c6668 to 60e7cdf Compare May 7, 2021 20:19

henryiii mentioned this pull request May 9, 2021

chore: bump macOS PyPy to 7.3.4, and drop PyPy 3.6 #666

Merged

henryiii force-pushed the feat/pypa-build branch 2 times, most recently from 3a77186 to 7b9b8e2 Compare May 17, 2021 21:45

henryiii mentioned this pull request May 18, 2021

chore: update to latest (manylinux1) requirements.txt pypy/manylinux#17

Open

henryiii force-pushed the feat/pypa-build branch from 7300e4a to 91230d5 Compare May 24, 2021 21:39

mayeut force-pushed the feat/pypa-build branch from d4d592e to 99dbec5 Compare May 25, 2021 16:48

henryiii mentioned this pull request May 25, 2021

Plan for 2.0 #657

Closed

12 tasks

mayeut force-pushed the feat/pypa-build branch from 99dbec5 to 094d7c9 Compare May 27, 2021 12:13

mayeut force-pushed the feat/pypa-build branch 2 times, most recently from 6f90957 to 592a270 Compare May 31, 2021 17:25

henryiii commented Jun 1, 2021

View reviewed changes

mayeut mentioned this pull request Jun 3, 2021

feat: configuration support #684

Merged

henryiii force-pushed the feat/pypa-build branch from 4c5368c to e2cf2a8 Compare June 21, 2021 19:24

refactor: use build-backend instead

f7d31fc

henryiii force-pushed the feat/pypa-build branch from e2cf2a8 to f7d31fc Compare June 22, 2021 02:55

henryiii marked this pull request as ready for review June 22, 2021 03:10

joerick reviewed Jun 22, 2021

View reviewed changes

henryiii and others added 3 commits June 22, 2021 10:27

Apply suggestions from code review

33d3a58

Co-authored-by: Joe Rickerby <[email protected]>

refactor: use literal types for build_frontend

b22f47c

refactor(tests): build_mode -> build_frontend_env

6c4d21a

joerick reviewed Jun 22, 2021

View reviewed changes

test/test_before_build.py Outdated Show resolved Hide resolved

Update test/test_before_build.py

a564d32

joerick approved these changes Jun 23, 2021

View reviewed changes

henryiii merged commit 515e9be into pypa:main Jun 23, 2021

joerick mentioned this pull request Jul 11, 2021

refactor: use run-like syntax #672

Closed

henryiii deleted the feat/pypa-build branch June 6, 2024 15:20

gitlab-importer mentioned this pull request Jun 21, 2021

venv fails on PyPy3.7 7.3.5 win64 when called from python.exe (symlink or copy to/of pypy3) pypy/pypy#3479

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support pypa build #521

feat: support pypa build #521

henryiii commented Jan 6, 2021

henryiii commented Jan 6, 2021

henryiii commented Jan 6, 2021 •

edited

Loading

henryiii commented Jan 6, 2021

joerick commented Jan 15, 2021

henryiii commented Mar 25, 2021

henryiii commented May 7, 2021

mayeut commented May 8, 2021

mayeut commented May 8, 2021

henryiii commented May 17, 2021

mayeut commented May 24, 2021

henryiii commented May 25, 2021

mayeut commented May 25, 2021

henryiii commented May 27, 2021

mayeut commented May 27, 2021

henryiii commented May 27, 2021

mayeut commented May 30, 2021

mayeut commented May 31, 2021 •

edited

Loading

mayeut commented May 31, 2021 •

edited

Loading

mayeut commented May 31, 2021

henryiii Jun 1, 2021

henryiii commented Jun 3, 2021

mayeut commented Jun 3, 2021

joerick commented Jun 3, 2021 •

edited

Loading

henryiii commented Jun 17, 2021

henryiii commented Jun 21, 2021

henryiii commented Jun 22, 2021

joerick left a comment

joerick commented Jun 22, 2021 •

edited by pradyunsg

Loading

henryiii commented Jun 23, 2021

		# we use shell=True here for windows, even though we don't need a shell due to a bug
		# https://bugs.python.org/issue8557

feat: support pypa build #521

feat: support pypa build #521

Conversation

henryiii commented Jan 6, 2021

henryiii commented Jan 6, 2021

henryiii commented Jan 6, 2021 • edited Loading

henryiii commented Jan 6, 2021

joerick commented Jan 15, 2021

henryiii commented Mar 25, 2021

henryiii commented May 7, 2021

mayeut commented May 8, 2021

mayeut commented May 8, 2021

henryiii commented May 17, 2021

mayeut commented May 24, 2021

henryiii commented May 25, 2021

mayeut commented May 25, 2021

henryiii commented May 27, 2021

mayeut commented May 27, 2021

henryiii commented May 27, 2021

mayeut commented May 30, 2021

mayeut commented May 31, 2021 • edited Loading

mayeut commented May 31, 2021 • edited Loading

mayeut commented May 31, 2021

henryiii Jun 1, 2021

Choose a reason for hiding this comment

henryiii commented Jun 3, 2021

mayeut commented Jun 3, 2021

joerick commented Jun 3, 2021 • edited Loading

henryiii commented Jun 17, 2021

henryiii commented Jun 21, 2021

henryiii commented Jun 22, 2021

joerick left a comment

Choose a reason for hiding this comment

joerick commented Jun 22, 2021 • edited by pradyunsg Loading

henryiii commented Jun 23, 2021

henryiii commented Jan 6, 2021 •

edited

Loading

mayeut commented May 31, 2021 •

edited

Loading

mayeut commented May 31, 2021 •

edited

Loading

joerick commented Jun 3, 2021 •

edited

Loading

joerick commented Jun 22, 2021 •

edited by pradyunsg

Loading