Manage python dependencies with setup.py/setuptools

[maconfr]

Using setup.py/setuptools allows us to manage dependencies with relaxed version requirements in the setup.py (if any) and using the requirements.txt to pin the packages and sub-packages to the exact versions that are garanteed to work.
This makes it easy to do updates and have reproducable environment due to the included and pinned sub-packages.

[leoluk]

ConfigMaster is an end-user server application and not a library, so I don't think it needs to (or should) be installable using setuptools/pip. If we ever wanted to include a setup.py, it would have to install the same dependencies (see https://github.com/getsentry/sentry/blob/master/setup.py for an example), or it would confuse people into using the wrong dependencies.

We already pin all packages and sub-packages and you can just run pip install --upgrade inside the virtualenv to upgrade, no?

[maconfr]

Managing packages in this way makes it difficult to distinguish between those that our project directly depends on and the sub-packages.
So how do we prevent us from dragging dependencies along forever even though we or the package we use no longer needs them?

This would be easy to solve with a list of direct dependencies in the setup.py and a requirements.txt with the pip freeze output.
Just create a fresh virtualenv, then execute 'pip install -e . && pip freeze > requirements.txt'.

Or is there a tool that is able to detect which packages are actually used and cleans up the environment/requirements.txt?

[leoluk]

Ah, I understand where you're coming from. Good point indeed. Still unconvinced that putting the unconstrained dependencies in setup.py is the proper solution (it's not a library, after all, and it's never going to be consumed as a package). I checked a few similar projects and everyone is doing it differently - guess Python packaging is still a mess.

I suggest we put the direct dependencies in requirements.txt with loose constraints, and have a separate requirements_lock.txt which pins all versions to the exact hash 1, making the builds fully reproducible.

There's pipfile 2 too, but it's not production-ready yet.

[maconfr]

The use of setup. py is of course not required to define our direct dependencies and generate a lock file including subdependencies for reproducible environments.
I am fine with your suggestion to do this with simple text files.
Maybe pip-tools could also help us to maintain our requirements. See: https://github.com/jazzband/pip-tools

However, I would like to point out the misunderstanding that setuptools is for "libraries". Setuptools describes a "project", regardless of its specific nature (except it is primarily a python project of course).
https://setuptools.readthedocs.io/en/latest/

But I agree, even though a lot has already improved, python packaging is still a mess.

[leoluk]

We'll also have to think about optional dependencies (django-auth-ldap, ...).