no hpy/__init__.py?

[arigo]

Is there a reason for why there is no hpy/__init__.py file, turning hpy into a "namespace"? Apparently that triggers a bug in "setup.py install" that seems not to handle this properly. I'd vote for avoiding namespaces if setuptools doesn't reliably support them.

[antocuni]

The original PR/discussion is this: #16

TL;DR: we need it to be a namespace if we want to distribute hpy.universal and hpy.devel (and possibly more) separately. I think there are already other projects using this structure.

What is the problem you are experiencing?

[arigo]

I didn't check but jtiai on irc said "looks like there is some kind of a bug in setup.py install".

[antocuni]

I'm not opposed to refactor/change things, but I'd like something more than that as a reason 😅

[arigo]

According to https://setuptools.readthedocs.io/en/latest/setuptools.html#namespace-packages, we're not doing the right thing. This docs says we must put an __init__.py, but one with a specific line in it (I didn't study this in detail).

[antocuni]

I think that that link refers to "pkg_resources-style", the "old" way of doing namespaces, which is compatible with Python2.
This link summarizes the possibilities. Indeed, it seems to imply that "native namespace packages" do no work with setup.py install, although I don't know the details.
For reference, see also PEP 420 for why they think native namespaces are a good idea.

So, I think we have two choices:

1. keep using native namespaces, and possibly emit a warning/error which tells you to use pip in case you use setup.py install (hint: you can always do pip install .)

2. switch to pkgutil-style namespaces.

That said, I am fine with either solution, but I suggest we should ask to someone who knows more about this to get all the implications.

[mattip]

@ncoghlan was one of the authors, maybe we could get a response or a referral to someone else who could help out.

[uranusjr]

Since setup.py install is strongly disfavoured, it is probably better to discourage it as much as possible.

This also bring out a related issue; HPy currently relies quite heavily on setuptools and distutils, which is an approach the packaging community is actively moving away from. This is not an issue for proof-of-concept works (where setup.py build_ext is really the only packaging command that needs to work), but would cause an issue when a more general audience look to try HPy out, where they’d want to install HPy via a wheel, and create PEP 517 wheels from extensions built with HPy, etc.

I’m currently pretty occupied myself, but intend to try some modern packaging strategies on HPy itself, and the extensions it builds. HPy is an awesome project, and this is the best I can do to help.

[mattip]

[I] intend to try some modern packaging strategies on HPy itself ...

That would be awesome. Are there models of modern packaging strategies for building c-extensions you would recommend looking at for inspiration?

[uranusjr]

Setuptools is actually a pretty good (the best?) choice for building the extension modules, since it contains a good amount of practically-working compiler and dependency introspection logic. Alternative build systems are unforauntately not very actively pursued afaik since setuptools.Extension is “good enough” for most people, which is not unreasonable.

The approach that may be beneficial is to create an interface to stand into setuptools.Extension, so other packaging tools can use the same interface to build both old- and new-style extension libraries. Pdm and Enscons have made some efforts interfacing with setuptools.Extension, but more work is needed.

[mattip]

Maybe we should take this offline, but just summarizing the rabbit hole your links sent me down:

• Enscons is a wrapper around SCons to create wheels, you specify it in a pyproject.toml. Although it claims to be "new style" it still requires setuptools. Is that to find the MSVC compiler, or does it need more?
• mesonpep517 wraps the meson build system. It seems a little bare-bones right now.
• scikit-build precedes PEP517 and monkey-patches setuptools to get itself into play

There are probably more

It is not clear to me how HPy would create an interface that would allow it to be used easily with all these tools. Here are a few examples of ways other projects interfacing compiled code have dealt with this:

• CFFI, provides a setuptools helper cffi_modules (described just above this link with an example)
• For Rust there is a setuptools RustExtension, there is an example in the link
• Cython (I think) precedes setuptools. There are many use patterns, like the canonical example from the docs, or this one

[antocuni]

@uranusjr thank you for your feedback, your help will surely be welcome.

However, I am not sure to understand what is the exact problem you are trying to solve (which doesn't mean that the problem does not exist of course). In test_pof.sh we test three different installation strategies:

1. setup.py bdist_wheel + pip install
   

   hpy/proof-of-concept/test_pof.sh

   Lines 72 to 89 in cc98cac

   	wheel() {
   	# build a wheel, install and test
   	HPY_ABI="$1"
   	VENV="venv/wheel_runner_$HPY_ABI"
   	clean
   	echo "=== testing setup.py bdist_wheel ==="
   	_build_wheel "$HPY_ABI"
   	WHEEL=`ls proof-of-concept/dist/*.whl`
   	echo "Wheel created: ${WHEEL}"
   	echo
   	echo "Create venv: $VENV"
   	python3 -m venv "$VENV"
   	source "$VENV/bin/activate"
   	echo "Installing wheel"
   	python3 -m pip install $WHEEL
   	echo
   	_test_pof
   	}

2. setup.py install
3. setup.py build_ext -i

If I understand it correctly (and I'm not sure), (1) is all we need for "modern" installation tools. The fact that we also support (2) and (3) is a convenience but should not harm anyone.

Another thing to keep in mind is that we want to make the transition from Python/C to HPy as easy as possible, and the vast majority of current extensions are written with setup.py. That's why we introduced hpy_ext_modules

[uranusjr]

I should have provided more context from the beginning 🙂 I wrote the initial comment mainly to answer the namespace module question (no, don’t bother with setup.py install), and did not expect to get into this much detail this quickly.

A common subtext around most recent development in Python packaging is to avoid the coupling (both build-time and run-time) to the distutils/setuptools stack. This is actually kind of analogous to what HPy aims to improve—there are certain things the default stack (CPython for HPy, distutils/setuptools for packaging) does not do well, and we want to provide alternatives to solve those issues. But the interface the default stack provides is not adequate for alternatives to implement, so we want to introduce better abstractions for people to more easily interface with multiple solutions.

HPy’s current build system has a hard dependency on distutils and setuptools, and interfaces with them in a way alternative packaging tools cannot reuse. This means those solutions cannot support building HPy extensions without rewriting most of the tool stack. So what I’m trying to solve is to look into possible improvements in this area early on, to avoid HPy becoming another thing that prevents people from using alternative packaging solutions.

[antocuni]

HPy’s current build system has a hard dependency on distutils and setuptools, and interfaces with them in a way alternative packaging tools cannot reuse. This means those solutions cannot support building HPy extensions without rewriting most of the tool stack. So what I’m trying to solve is to look into possible improvements in this area early on, to avoid HPy becoming another thing that prevents people from using alternative packaging solutions.

sounds good to me. Note that it's already partly like that, because most of the logic is inside hpy.devel.HPyDevel, which is used by both the testing infrastructure and the setuptools integration. However, HPyDevel is based on setuptools, so you probably want to decouple the logic there.

Feel free to come to the IRC channel and/or write to the mailing list and/or open an issue to discuss more concrete details.

[cclauss]

#63 (comment)

The fact that we also support (2) and (3) is a convenience but should not harm anyone.

It seems to break the GitHub Actions (see #467) and harms users on macOS v12, v13, v14.