Switch to Poetry

[Conan-Kudo]

This allows kiwi to be built, developed, tested, and installed through Poetry.

Fixes #2370.
Fixes #2369.

Changes proposed in this pull request:

• Add Poetry definition
• Remove setuptools definition
• Raise the minimum Python version to 3.9.

[Conan-Kudo]

This is nowhere near complete, because I'm still working on figuring out:

• Converting our optional dependencies into extras
• Handling our special sdist logic (if at all possible)
• Reworking our CI to use it
• Reworking our spec to use it
• Update contribution chapter in the docs
• Handle bumpversion properly?

[schaefi]

Thanks for starting this. I switched the branch and did some tests. I wondered about the entry-points. Usually I do this:

tox -e devel
. .tox/unit_py3_11/bin/activate
which kiwi-ng
/home/ms/Project/kiwi/.tox/unit_py3_11/bin/kiwi-ng
kiwi-ng --version
KIWI (next generation) version 10.0.0
deactivate

now the same with poetry

poetry shell
Spawning shell within /home/ms/.cache/pypoetry/virtualenvs/kiwi-Btua-i95-py3.11
. /home/ms/.cache/pypoetry/virtualenvs/kiwi-Btua-i95-py3.11/bin/activate
which python3
/home/ms/.cache/pypoetry/virtualenvs/kiwi-Btua-i95-py3.11/bin/python3
==> ok
which kiwi-ng
which: no kiwi-ng in ...
==> Hmm, no entry points created
exit

I wonder how you would call kiwi-ng inside of the venv without the entry-points. The entry-points are defined in setup.py as

'entry_points': {
        'kiwi.tasks': [
            'image_info=kiwi.tasks.image_info',
            'image_resize=kiwi.tasks.image_resize',
            'result_bundle=kiwi.tasks.result_bundle',
            'result_list=kiwi.tasks.result_list',
            'system_build=kiwi.tasks.system_build',
            'system_create=kiwi.tasks.system_create',
            'system_prepare=kiwi.tasks.system_prepare',
            'system_update=kiwi.tasks.system_update'
        ],
        'console_scripts': [
            'kiwi-ng=kiwi.kiwi:main',
            'kiwicompat=kiwi.kiwi_compat:main'
        ]
    },

It's really crucial that this stuff continues to work. Not only for the usability of a venv but also to not break kiwi's plugin support

[schaefi]

btw the kiwicompat entry point can be deleted, we won't need any old compat layer for 10.x.x

[Conan-Kudo]

With poetry, you initialize the virtualenv using poetry itself, and use poetry run to run the script.

[schaefi]

With poetry, you initialize the virtualenv using poetry itself, and use poetry run to run the script.

ah I see

poetry install
poetry run kiwi-ng --version
KIWI (next generation) version 10.0.0

works great, thanks

And thanks for adding the plugin related data 👍

[schaefi]

There is also a contribution chapter in doc/source/contributing.rst. That information needs to be adapted as it will become obsolete by the changes

[dcermak]

No poetry.lock?

[dcermak]

@Conan-Kudo I was wondering what you think of https://hatch.pypa.io/latest/why/? I have no experience with it, but it appears like a lighter and more upstream aligned build system. WDYT?

[Conan-Kudo]

PyPA disclaims "upstream-alignment" in general, especially after pipx. Mainline Python has been trying to be less opinionated over time about how you build stuff as long as it implements the PEP 517/518 interface.

[Conan-Kudo]

So I think the last thing to figure out is bumpversion...

[schaefi]

unresolvable: nothing provides python3-sphinx

[schaefi]

And many python311-xxx unresolvables

[schaefi]

unresolvable: nothing provides python3-installer on some ubuntu targets

[Conan-Kudo]

And many python311-xxx unresolvables

We need openSUSE Leap 15.6 target enabled for python311-xxx stuff.

[schaefi]

And many python311-xxx unresolvables

We need openSUSE Leap 15.6 target enabled for python311-xxx stuff.

yes did that, and enabled backports repo for older Leap. So we are down to that sphinx requirement which was also not found for other targets

[schaefi]

All other package builds failed because of

[   40s] RPM build errors:
[   40s]     File not found: /home/abuild/rpmbuild/BUILDROOT/python-kiwi-10.0.0-39.1.x86_64/usr/bin/kiwicompat

we dropped that, it seems to be still in the package list

[Conan-Kudo]

working on those now, and also fixing the PKGBUILD

[schaefi]

btw, why was sphinx added to be a package dependency ? It's imho only a development dependency. That's also why didn't saw this before

[Conan-Kudo]

btw, why was sphinx added to be a package dependency ? It's imho only a development dependency. That's also why didn't saw this before

It's required to build the man pages.

[schaefi]

It's required to build the man pages.

ah so we render the man pages now at package build time. I think before we rendered them with the "make build" target and put them into the tarball. ok your way is better but we have new buildrequires now

[schaefi]

if we don't find a solution we can also render the man pages via rst2man

for man in $(shell find -name "*.rst" | sed -e "[email protected]@@"); do \
        rst2man $$man.rst $$man.8 ;\
    done

I do it that way in another project

[Conan-Kudo]

It's because it's python3-Sphinx in SUSE distributions, easy fix. Please refresh staging.

[schaefi]

ah ok, I will

[schaefi]

done, less unresolvables more errors :)

[   14s] sphinx-build -b man -W -d build/doctrees   source build/man
[   14s] Running Sphinx v7.2.6
[   15s] [autosummary] generating autosummary for: api.rst, api/kiwi.archive.rst, api/kiwi.boot.image.rst, api/kiwi.boot.rst, api/kiwi.bootloader.config.rst, api/kiwi.bootloader.install.rst, api/kiwi.bootloader.rst, api/kiwi.bootloader.template.rst, api/kiwi.builder.rst, api/kiwi.container.rst, ..., working_with_images/disk_setup_for_vagrant.rst, working_with_images/iso_to_usb_stick_deployment.rst, working_with_images/iso_to_usb_stick_file_based_deployment.rst, working_with_images/iso_to_usb_stick_grub2_boot_from_iso.rst, working_with_images/legacy_netboot_root_filesystem.rst, working_with_images/network_live_iso_boot.rst, working_with_images/network_overlay_boot.rst, working_with_images/setup_network_bootserver.rst, working_with_images/setup_yast_on_first_boot.rst, working_with_images/use_suse_media.rst
[   15s] 
[   15s] Warning, treated as error:
[   15s] Failed to import kiwi.system.prepare.
[   15s] Possible hints:
[   15s] * AttributeError: module 'kiwi.system' has no attribute 'prepare'
[   15s] * ModuleNotFoundError: No module named 'pkg_resources'
[   15s] make: *** [Makefile:157: man] Error 2

[schaefi]

I'm out for today. I added you as admin to Virtualization:Appliances:Staging/python-kiwi so you can eat your own dog food

make build
cp dist/* ...
osc ci

[Conan-Kudo]

Everything is now fixed up in https://build.opensuse.org/project/show/Virtualization:Appliances:Staging

[schaefi]

Thanks @Conan-Kudo, I went through a night session to fixup all integration tests due to the shift of some of them into newer distros. Ubuntu tests are based on Lunar now, Debian tests are based on Debian 12, Leap uses backports as the kiwi package there builds for py311.

In a final change here I fixed the python-kiwi-pkgbuild-template to also apply the docopt-ng vs. docopt workaround.
After that all integration tests builds and I will now merge this PR

One issue is the publish to pypi and the publishing of the docs. This we will find out when I try the release, so maybe there will be a hiccup when we release