Skip to content

AiiDA Dependency Management

Jump to bottom
Daniel Hollas edited this page Nov 18, 2024 · 44 revisions

test-install

About

This page contains guidelines, concrete procedures, and any other information relevant to the management of dependencies for the aiida-core package and plugins.

General information

The dependencies for the aiida-core package must be managed according to AEP 002.

Dependency Manager (DM):

The current dependency manager should be a member of the dependency-manager team.

Specification of AiiDA core dependencies

The dependencies for the aiida-core package are specified in the pyproject.toml file are divided into main dependencies in dependencies list and optional dependencies (extras) in project.optional-dependencies table.

Furthermore, dependencies are also specified in

  • environment.yml: to create a conda environment
  • requirements/requirements-py-3x.txt: well-defined Python environments for all supported Python versions in compliance with dependencies used for CI testing; these files are automatically maintained by the 'update-requirements' workflow

The specification of a dependency within the pyproject.toml is to be considered authoritative.

In addition to above mentioned files, the DM must also be concerned with the modification of the following files:

  • utils/dependency_management.py: Utility script used for dependency management related tasks.
  • .github/workflows/ci.yml: Specifies the CI workflow that checks consistency for dependency-specifying files (pyproject.toml, requirements/*.txt, etc.) and carries out tests for environments specified in requirements/*.txt files.
  • .github/workflows/test-install.yml: Specifies the a workflow that tests for the ability to install aiida-core with the given dependencies within the current ecosystem. It also defines a workflow for the semi-automated update of the requirements/*.txt files as part of a pull request. Is executed upon modification of dependencies (pyproject.toml) and nightly.

Utility script

The utils/dependency_management.py script within the aiida-core repository provides commands to validate the consistency of aforementioned files with the pyproject.toml file as well as commands to re-generate them after updating a dependency.

How to modify the dependency specification

Prerequisites

Before adding a new dependency for the aiida-core package, make sure that the dependency adheres to the following requirements:

The newly introduced dependency

  • is needed to close a non-trivial feature gap that could not be resolved easily otherwise,
  • supports all Python versions supported by aiida-core (as specified in pyproject.toml),
  • is available on PyPI and conda-forge [1],
  • appears to be within a stable development stage, e.g., has reached a version 1.0 for projects that follow semantic versioning.
  • uses an MIT-compatible license (e.g. MIT, BSD, Apache, LGPL but not GPL)

[1] In case that a dependency is not yet available through aforementioned channels, but is considered critical enough to be included anyways, the following steps can be taken:

  1. For lightweight dependencies, consider to vendor the package if permissible by license.
  2. Create a request to the current maintainer to make the dependency available open-source/on PyPI/on conda-forge.
  3. Maintain the PyPI project and conda-forge recipes ourselves.

Flow for adding/updating dependencies

This is the standard workflow for updating the aiida-core dependencies:

1. Update the pyproject.toml file:

Modify the affected entries in pyproject.toml and then update all dependent files with

./util/dependency_management.py generate-all

The command will also be executed by the pre-commit hook (if installed).

For packages that are named differently between PyPI and conda-forge, you might need to add an entry to the SETUPTOOLS_CONDA_MAPPINGS variable within the same script.

2. Update the requirements/ files

The requirements/requirements-py-3.?.txt files define the Python pinned environments used for testing and must be consistent with the dependency specification in the pyproject.toml file. There are two ways you can update those files: semi-automatically and manually.

For a semi-automated update (recommended), push the branch that contains the updated dependency directly to the aiida-core package and create a pull request. The GitHub actions bot will then attempt to create a second pull request with the updated requirements/* files into that same branch. Note: This will not work for pull requests from forks.

For a manual update, simply modify the affected entry within all requirements/ files such that they are consistent with the dependency specification, can be installed for all supported Python versions, and do not create any dependency conflicts. For example, if you have a dependency foo~=1.5.0, then the corresponding entry in the requirements file could be foo==1.5.6 or foo==1.7.1, but not foo==1.4.8 or foo==2.0.1.

Continuous Integration

The consistency of the various files with pyproject.toml is checked with a pre-commit hook that is executing the generate-all and the validate-* commands of utils/dependency_management.py. This hook is also executed for all commits as part of the aiida-core CI workflow.

Any changes to pyproject.toml file also trigger the test-install workflow, that explicitly checks whether the package can be installed with pip and conda, and whether the requirements/ files are consistent with the dependency specification. In addition, the test-install workflow is executed nightly to ensure that changes within the Python ecosystem do not lead to issues with installation or cause tests to fail. Finally, the test--install workflow will also attempt to semi-automatically update the requirements/*.txt files in case that inconsistencies are detected by creating a separate pull request into the branch that modifies the dependency specification.

Constraint dependencies

The set of dependencies that are currently constrained and where we should make an effort towards loosening them is maintained in the form of GitHub issues with the topic/dependenciesc/constraint label.

Problematic dependencies

Note: The following list of problematic dependencies should be considered outdated as of 2021-08-30.

This is a list of dependencies that might be problematic for some reason.

Impact on the build and install chain of aiida-core:

  1. Critical disruption/blocker
  2. Major disruption with user impact
  3. Minor disruption without user impact
  4. Concern, but so far no impact.
  • [1] tornado: issue; old major version <5 required, causing issues for example in the context of AiiDA lab
  • [2] pymatgen: Their use of specifying numpy as a requirement in setup_requires in the call to setuptools.setup() is causing issues, because it will trigger an installation (and eventual build!) of numpy within the build environment for pymatgen, which will fail on Python 3.5 with a version of setuptools<42.0.0 and is suspected to cause other potential problems in the future.
  • [2] ase: Released version that is incompatible with Python 3.5, but not declared as such (see issue #4295).
  • [3] pyyaml: constrained to ~=5.1.2 due to issue #3709; causing CI issues in combination with Python version 3.7.8 (#4256)
  • [3] circus: Extremely long turn-around on a PR from us to circus in the context of resolving issues with the tornado dependency is raising concerns about how well circus is currently maintained (blocking).
  • [4] shortuiid: A broken release (0.5.1) was removed from PyPI after upload.
  • [4] frozendict: project abandoned; will likely not support Python 3.10, see discussion here.
  • [4] plumpy: Will require update for the support for Python 3.10 (related issue).
Clone this wiki locally