CI: Utilize uv lockfile for reproducible test environments

[danielhollas]

This PR removes the requirement files in requirements/ folder and replaces them with a single universal lockfile uv.lock. "universal" in this case means that it contains resolved package versions for all supported python versions and operating systems. More on the uv lockfiles can be found in their docs: https://docs.astral.sh/uv/concepts/projects/layout/#the-lockfile

When modifying the dependencies in pyproject.toml, the lockfile needs to be updated by running

uv lock

(Note that using e.g. uv add, uv remove or uv sync updates the lockfile automatically)

This relative simplicity compared to the previous custom solution allows for much easier development and I could delete a lot of custom YAML in CI. To check whether the lockfile is up to date, one can simply run

uv lock --locked

This command is run automatically as a pre-commit hook whenever pyproject.toml is changed.

Note

Developers don't necessarily need to use the lockfile locally, nor are they required to have uv installed, unless they need to modify dependencies in pyproject.toml

[danielhollas]

NOTE: By default we're installing with pre-commit optional dependencies (extras), which transitively includes other extras, see pyproject.toml I think this is the best solution to make things work for now.

[danielhollas]

It looks like @aiidateam/dependency-manager doesn't exist? Should I assign @agoscinski explicitly?

[unkcpz]

Maybe add both me and @agoscinski, or if you also want to maintain this ;) I think after this PR, the dependency management can be future simplified.

[danielhollas]

This file does not exist.

[danielhollas]

@agoscinski @unkcpz I think this PR is mostly ready now, let me know what you think!

The only thing missing is developer documentation: I would suggest moving the Dependency Management Wiki article to the repo in a separate PR.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.94%. Comparing base (ef60b66) to head (10cd62a).
Report is 146 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6640      +/-   ##
==========================================
+ Coverage   77.51%   77.94%   +0.44%     
==========================================
  Files         560      563       +3     
  Lines       41444    41659     +215     
==========================================
+ Hits        32120    32466     +346     
+ Misses       9324     9193     -131     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[unkcpz]

The only thing missing is developer documentation: I would suggest moving the Dependency Management Wiki article to the repo in a separate PR.

Sure, go ahead with a separate PR, thanks!

[khsrali]

@danielhollas nice, thanks a lot!
suppose I have an entry in my pyproject.toml as such:
'plumpy@git+https://github.com/khsrali/plumpy.git@allow-async-upload-download#egg=plumpy',

would this be understood and accepted by uv lock?

[danielhollas]

@khsrali I tried it and it works without a problem. :-)
btw: I don't think you need the #egg=plumpy part at the and of the URL, and uv strips it anyways.

[unkcpz]

unless they need to modify dependencies in pyproject.toml

I don't understand how this will work. I think if we run uv lock from different machine, the lock file is generated using the environment where the command is run. Which means two people will have two different lock file generated. Is this leading to large number of lines change?

[unkcpz]

Where is this .venv created from?

[danielhollas]

It is created by uv sync automatically in the install-aiida-core action above.

[danielhollas]

I think if we run uv lock from different machine, the lock file is generated using the environment where the command is run.

uv should generate a universal lock file independent of your current python version or OS. At least that is my understanding, would be good if you can verify. I have generated the lock here on Linux and Python 3.12

Also you can try using the lock file I generated by running uv sync. It should work regardless of your python version.