setup.cfg converter

[chobeat]

Hello people,

in my company we are working on a setup.cfg conversion for pyscaffold to migrate a bunch of projects from the standard pyscaffold structure to Poetry. This means migrating all the metadata, dependencies and entry points (possibly also other stuff we have missed) to pyproject.toml.

I guess this might be an interesting contribution to put here. For what I see the project currently doesn't do much to fully support Poetry, right? Or is there some magic I'm missing?

[FlorianWilhelm]

Hi @chobeat,

can you name the reasons for the switch to Poetry? I know that it also includes the handling of the dependencies but besides that?

PyScaffold uses setuptools which handles things totally differently than Poetry and thus we cannot support it. Also with Poetry there is actually no need for a tool like PyScaffold, just do poetry new project instead of putup project.

As a maintainer of PyScaffold I am biased of course but here are the downsides I see with Poetry: The management of your virtual environment dependencies and the actual development and packaging are too much intertwined for my taste. How would you, for instance, use Conda packages if you want to use rather those instead of pip? Also the project setup with new is extremely basic compared to what you get if you use PyScaffold. The much recommended src layout is also not used. Poetry doesn't seem to support extensions etc.

For these reasons, be sure you migrate all your projects to Poetry for the right reasons and not only because it's bleeding-edge and has a fancy website. Regarding your suggestion of a contribution for converting a PyScaffold project to Poetry, for obvious reasons we have no interest in promoting tools that drive people away from PyScaffold since we are still convinced it's currently the best solution in the Python ecosystem to get a complete Python project scaffold with everything set up for professional development. For these reasons, thank you for your suggestion but rather create a separate repository outside the PyScaffold organization for your conversion tool.

[chobeat]

I think you misunderstood what I proposed. In my company and in my vision, PyScaffold and Poetry coexist with different roles. In my mind, this tool would simply allow the user to replicate a project definition (in setup.cfg) to a pyproject.toml format in order to use poetry for those operations that are more convoluted with the normal python toolchain, in particular publishing.

Anyway, internally, we are migrating for continued issues with dependency resolution and environment management that are too tricky to solve with setuptools and distutils, because they are not very transparent tools.

This migration won't mean we will get rid of pyscaffold, quite the contrary. With it we also consolidated the role of PyScaffold in our organization, basically with a much more serious and solid development of our custom pyscaffold extension. Especially the extension part of pyscaffold remains central in the way we use PyScaffold because it allows us to support many scenarios. For example we added custom Sonar configurations, custom TeamCity run scripts for each python module and so on and so forth. Also, to maintain compatibility with older dockerfiles (at least temporarily), we maintain both a full setup.cfg and pyproject.toml file. This is something we wil get rid of, leaving to the setup.cfg the centralization of the configurations that don't fit into the pyproject.toml, like flake8 or pytest.

So the "scaffolding" part of poetry is not contemplated here, poetry new is never used. PyScaffold, for a new project, will be the tool used to do all the work and will allow the user to do stuff like poetry install or poetry run ... instead of the usual mess of virtualenvs, setuptools, distutils, pip and so on.

[FlorianWilhelm]

Hi @chobeat, okay, in that case, a pyscaffoldext-poetry extension would make sense I guess. It should then provide some additional command-line tool that after a change of setupt.cfg converts those changes into the appropriate section in pyproject.toml. Is that would you mean? Sorry, that I didn't get your suggestion right the first time.

[chobeat]

I think it should behave like a normal extension, where the only output is the pyproject.toml file in the root of the project. The only option I would put is a --no-duplicate-config to remove the original sections and options in the setup.cfg file (like metadata and dependencies) but keep the rest there (flake8, pytest...).

During the creation of a new project this is quite straightforward. During an update, we would like to load the existing keys to transform them and put them into pyproject. For what I know, this is something that doesn't really fit into the abstraction of pyscaffold and in our extension we just load the file from the expected location (a bit dirty but better than nothing). So in case of upgrade we might overwrite the part of the setup.cfg if --no-duplicate-config is activated. But these are all thing we can agree on later, most of the complexity is to fully support the setup.cfg format. Probably it's worth to package this Converter into its own class because it might be useful to other people and one day we might want to release it as an independent piece.

That said, what additional command-line tool are you talking about? It's not very clear.

[FlorianWilhelm]

Okay, now I understood some more. All the meta-information is managed in pyproject.toml and removed in setup.cfg. I thought one changes only setup.cfg and after that runs every time a conversion tool that updates the meta information in pyproject.toml. This way one would always have both options to package. But I am also fine if one edits only pyproject.toml for meta-information also long as there is no need for double bookkeeping.

How would you integrate the features of setuptools-scm into pyproject.toml meaning the automatic version determination based on git? I think also the definition of __version__ in __init__.py would need some change in the case of poetry.

Feel free to create a repo like pyscaffoldext-poetry within the PyScaffold organization and try out things.

[chobeat]

The version is a problem. Now we just run this outside of poetry in the release script. I might try some magic with the build option of poetry. I know about this thread: python-poetry/poetry#140 but there's nothing definitive.

Anyway in the coming weeks I will try to bootstrap the work and move what we did to github.

[bealdav]

Hi @chobeat any news about your tool ?