🔧 Move to flit for PEP 621 compliant package build (aiidateam#5312)

The build process for python packages has recently been standardised,
primarily via [PEP 621](https://www.python.org/dev/peps/pep-0621)
(`pyproject.toml` only build specification),
and [PEP 660](https://www.python.org/dev/peps/pep-0660)
(editable installs via `pyproject.toml`).

https://github.com/pypa/flit is a build tool, for pure-python packages,
compliant with both of these PEPs.
Switching to this build tool allows for the contents of the `setuptools` specific
`setup.json`, `setup.py` and `MANIFEST.in` files
to be transposed in to `pyproject.toml` and removed.

The [fastentrypoints](https://github.com/ninjaaron/fast-entry_points)
build requirement has been removed,
since with flit the issue it solves is no longer present.

The package version is now sourced directly from `aiida/__init__.py`,
and so there is no need for any consistency checks, which have been removed.

This change also removes the dynamically generated `all` extra,
and the logic to append `rest`+`atomic_tools` to the `tests` and `docs` extras,
present in the `setup.py`.

Note that `pip install --editable .`, requires pip version>=21 for PEP 660 compliance.

.github/workflows/check_release_tag.py

@@ -1,16 +1,31 @@
 # -*- coding: utf-8 -*-
 """Check that the GitHub release tag matches the package version."""
 import argparse
-import json
+import ast
+from pathlib import Path
+
+
+def get_version_from_module(content: str) -> str:
+    """Get the __version__ value from a module."""
+    # adapted from setuptools/config.py
+    try:
+        module = ast.parse(content)
+    except SyntaxError as exc:
+        raise IOError(f'Unable to parse module: {exc}')
+    try:
+        return next(
+            ast.literal_eval(statement.value) for statement in module.body if isinstance(statement, ast.Assign)
+            for target in statement.targets if isinstance(target, ast.Name) and target.id == '__version__'
+        )
+    except StopIteration:
+        raise IOError('Unable to find __version__ in module')
+
 
 if __name__ == '__main__':
     parser = argparse.ArgumentParser()
     parser.add_argument('GITHUB_REF', help='The GITHUB_REF environmental variable')
-    parser.add_argument('SETUP_PATH', help='Path to the setup.json')
     args = parser.parse_args()
     assert args.GITHUB_REF.startswith('refs/tags/v'), f'GITHUB_REF should start with "refs/tags/v": {args.GITHUB_REF}'
     tag_version = args.GITHUB_REF[11:]
-    with open(args.SETUP_PATH, encoding='utf8') as handle:
-        data = json.load(handle)
-    pypi_version = data['version']
-    assert tag_version == pypi_version, f'The tag version {tag_version} != {pypi_version} specified in `setup.json`'
+    pypi_version = get_version_from_module(Path('aiida/__init__.py').read_text(encoding='utf-8'))
+    assert tag_version == pypi_version, f'The tag version {tag_version} != {pypi_version} specified in `pyproject.toml`'

.github/workflows/ci-code.yml

@@ -33,7 +33,7 @@ jobs:
       if: failure() && steps.check_reqs.outputs.error
       uses: peter-evans/commit-comment@v1
       with:
-        path: setup.json
+        path: pyproject.toml
         body: |
           ${{ steps.check_reqs.outputs.error }}
 

.github/workflows/ci-style.yml

@@ -29,6 +29,7 @@ jobs:
 
     - name: Install python dependencies
       run: |
+        pip install --upgrade pip
         pip install -r requirements/requirements-py-3.8.txt
         pip install -e .[pre-commit]
         pip freeze

.github/workflows/docs-build.yml

@@ -22,7 +22,8 @@ jobs:
         python-version: '3.8'
     - name: Install python dependencies
       run: |
-        pip install -e .[docs,tests]
+        pip install --upgrade pip
+        pip install -e .[docs,tests,rest,atomic_tools]
     - name: Build HTML docs
       id: linkcheck
       run: |

.github/workflows/post-release.yml

@@ -30,8 +30,9 @@ jobs:
 
     - name: Install python dependencies
       run: |
+        pip install --upgrade pip
         pip install transifex-client sphinx-intl
-        pip install -e .[docs,tests]
+        pip install -e .[docs,tests,rest,atomic_tools]
 
     - name: Build pot files
       env:

.github/workflows/release.yml

@@ -23,7 +23,7 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: '3.8'
-    - run: python .github/workflows/check_release_tag.py $GITHUB_REF setup.json
+    - run: python .github/workflows/check_release_tag.py $GITHUB_REF
 
   pre-commit:
 
@@ -43,7 +43,11 @@ jobs:
         sudo apt update
         sudo apt install libkrb5-dev ruby ruby-dev
     - name: Install python dependencies
-      run: pip install -e .[all]
+      run: |
+        pip install --upgrade pip
+        pip install -r requirements/requirements-py-3.8.txt
+        pip install -e .[pre-commit]
+        pip freeze
     - name: Run pre-commit
       run: pre-commit run --all-files || ( git status --short ; git diff ; exit 1 )
 
@@ -126,12 +130,12 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: '3.8'
-    - name: Build package
+    - name: install flit
       run: |
-        pip install wheel
-        python setup.py sdist bdist_wheel
-    - name: Publish to PyPI
-      uses: pypa/gh-action-pypi-publish@v1.1.0
-      with:
-        user: __token__
-        password: ${{ secrets.PYPI_KEY }}
+        pip install flit~=3.4
+    - name: Build and publish
+      run: |
+        flit publish
+      env:
+        FLIT_USERNAME: __token__
+        FLIT_PASSWORD: ${{ secrets.PYPI_KEY }}

.github/workflows/test-install.yml

@@ -3,7 +3,6 @@ name: test-install
 on:
   pull_request:
     paths:
-      - 'setup.*'
       - 'environment.yml'
       - '**/requirements*.txt'
       - 'pyproject.toml'
@@ -251,14 +250,14 @@ jobs:
 
     # Add python-version specific requirements/ file to the requirements.txt artifact.
     # This artifact can be used in the next step to automatically create a pull request
-    # updating the requirements (in case they are  inconsistent with the setup.json file).
+    # updating the requirements (in case they are  inconsistent with the pyproject.toml file).
     - uses: actions/upload-artifact@v1
       if: matrix.backend == 'django'  # The requirements are identical between backends.
       with:
         name: requirements.txt
         path: requirements-py-${{ matrix.python-version }}.txt
 
-# Check whether the requirements/ files are consistent with the dependency specification in the setup.json file.
+# Check whether the requirements/ files are consistent with the dependency specification in the pyproject.toml file.
 # If the check fails, warn the user via a comment and try to automatically create a pull request to update the files
 # (does not work on pull requests from forks).
 
@@ -293,7 +292,7 @@ jobs:
       uses: peter-evans/commit-comment@v1
       with:
         token: ${{ secrets.GITHUB_TOKEN }}
-        path: setup.json
+        path: pyproject.toml
         body: |
           The requirements/ files are inconsistent!
 
@@ -328,7 +327,7 @@ jobs:
         title: "Update requirements/ files."
         body: |
           Update requirements files to ensure that they are consistent
-          with the dependencies specified in the 'setup.json' file.
+          with the dependencies specified in the 'pyproject.toml' file.
 
           Please note, that this pull request was likely created to
           resolve the inconsistency for a specific dependency, however
@@ -344,7 +343,7 @@ jobs:
         issue-number: ${{ github.event.number }}
         body: |
           I automatically created a pull request (#${{ steps.create_update_requirements_pr.outputs.pr_number }}) that adapts the
-          requirements/ files according to the dependencies specified in the 'setup.json' file.
+          requirements/ files according to the dependencies specified in the 'pyproject.toml' file.
 
     - name: Create PR comment on failure
       if: steps.create_update_requirements_pr.outcome == 'Failure'
@@ -353,4 +352,4 @@ jobs:
         issue-number: ${{ github.event.number }}
         body: |
           Please update the requirements/ files to ensure that they
-          are consistent with the dependencies specified in the 'setup.json' file.
+          are consistent with the dependencies specified in the 'pyproject.toml' file.

.pre-commit-config.yaml

@@ -1,7 +1,7 @@
 ci:
     autoupdate_schedule: monthly
     autofix_prs: true
-    skip: [mypy, pylint, dm-generate-all, dependencies, verdi-autodocs, version-number]
+    skip: [mypy, pylint, dm-generate-all, dependencies, verdi-autodocs]
 
 repos:
 -   repo: https://github.com/pre-commit/pre-commit-hooks
@@ -123,8 +123,7 @@ repos:
         pass_filenames: false
         files: >-
             (?x)^(
-                setup.py|
-                setup.json|
+                pyproject.toml|
                 utils/dependency_management.py
             )$
 
@@ -135,8 +134,7 @@ repos:
         pass_filenames: false
         files: >-
             (?x)^(
-                setup.json|
-                setup.py|
+                pyproject.toml|
                 utils/dependency_management.py|
                 environment.yml|
             )$
@@ -153,15 +151,3 @@ repos:
                 aiida/cmdline/params/types/.*|
                 utils/validate_consistency.py|
             )$
-
-    -   id: version-number
-        name: Check version numbers
-        entry: python ./utils/validate_consistency.py version
-        language: system
-        pass_filenames: false
-        files: >-
-            (?x)^(
-                setup.json|
-                utils/validate_consistency.py|
-                aiida/__init__.py
-            )$

.readthedocs.yml

@@ -15,6 +15,8 @@ python:
           extra_requirements:
             - docs
             - tests
+            - rest
+            - atomic_tools
 
 # Let the build fail if there are any warnings
 sphinx:

aiida/calculations/diff_tutorial/calculations.py

@@ -2,7 +2,7 @@
 """
 Calculations provided by aiida_diff tutorial plugin.
 
-Register calculations via the "aiida.calculations" entry point in the setup.json file.
+Register calculations via the "aiida.calculations" entry point in the pyproject.toml file.
 """
 from aiida.common import datastructures
 from aiida.engine import CalcJob

aiida/parsers/plugins/diff_tutorial/parsers.py

@@ -2,7 +2,7 @@
 """
 Parsers for DiffCalculation of plugin tutorial.
 
-Register parsers via the "aiida.parsers" entry point in the setup.json file.
+Register parsers via the "aiida.parsers" entry point in the pyproject.toml file.
 """
 # START PARSER HEAD
 from aiida.engine import ExitCode

docs/source/howto/plugins_develop.rst

@@ -84,17 +84,15 @@ Here is an example of a folder structure for an AiiDA plugin, illustrating diffe
       LICENSE              - license of your plugin
       MANIFEST.in          - lists non-python files to be installed, such as LICENSE
       README.md            - project description for github and PyPI
-      setup.json           - plugin metadata: installation requirements, author, entry points, etc.
-      setup.py             - PyPI installation script, parses setup.json and README.md
+      pyproject.toml       - plugin metadata: installation requirements, author, entry points, etc.
       ...
 
 A minimal plugin package instead might look like::
 
    aiida-minimal/
       aiida_minimal/
          __init__.py
-      setup.py
-      setup.json
+      pyproject.toml
 
 .. _how-to:plugins-develop:entrypoints:
 
@@ -111,14 +109,11 @@ Adding a new entry point consists of the following steps:
  #. Finding the right entry point group. You can list the entry point groups defined by AiiDA via ``verdi plugin list``.
     For a documentation of the groups, see :ref:`topics:plugins:entrypointgroups`.
 
- #. Adding the entry point to the ``entry_points`` field in the ``setup.json`` file::
+ #. Adding the entry point to the ``entry_points`` field in the ``pyproject.toml`` file::
 
      ...
-     entry_points={
-       "aiida.calculations": [
-         "mycode.<something> = aiida_mycode.calcs.some:MysomethingCalculation"
-       ]
-     }
+     [project.entry-points."aiida.calculations"]
+     "mycode.<something>" = "aiida_mycode.calcs.some:MysomethingCalculation"
      ...
 
 Your new entry point should now show up in ``verdi plugin list aiida.calculations``.
@@ -227,7 +222,7 @@ Since the source code of most AiiDA plugins is hosted on GitHub, the first conta
 
  * Make sure to have a useful ``README.md``, describing what your plugin does and how to install it.
  * Leaving a contact email and adding a license is also a good idea.
- * Make sure the information in the ``setup.json`` file is correct and up to date (in particular the version number), since this information is used to advertise your package on the AiiDA plugin registry.
+ * Make sure the information in the ``pyproject.toml`` file is correct and up to date (in particular the version number), since this information is used to advertise your package on the AiiDA plugin registry.
 
 Source-code-level documentation
 -------------------------------
@@ -279,8 +274,7 @@ AiiDA plugin packages are published on the `AiiDA plugin registry <registry_>`_
 
 Before publishing your plugin, make sure your plugin comes with:
 
- * a ``setup.json`` file with the plugin metadata
- * a ``setup.py`` file for installing your plugin via ``pip``
+ * a ``pyproject.toml`` file with the plugin metadata and for installing your plugin via ``pip``
  * a license
 
 For examples of these files, see the `aiida-diff demo plugin <aiida-diff_>`_.
@@ -297,7 +291,7 @@ In order to register your plugin package, simply go to the `plugin registry <reg
 
 .. note::
 
-  The plugin registry reads the metadata of your plugin from the ``setup.json`` file in your plugin repository.
+  The plugin registry reads the metadata of your plugin from the ``pyproject.toml`` file in your plugin repository.
 
 
 We encourage you to **get your plugin package listed as soon as possible**, both in order to reserve the plugin name and to inform others of the ongoing development.
@@ -308,10 +302,6 @@ Publishing on PyPI
 For distributing AiiDA plugin packages, we recommend to follow the `guidelines for packaging python projects <packaging_>`_, which include making the plugin available on the `python package index <PyPI_>`_.
 This makes it possible for users to simply ``pip install aiida-myplugin``.
 
-.. note::
-    When updating the version of your plugin, don't forget to update the version number both in the ``setup.json`` and in ``aiida_mycode/__init__.py``.
-
-
 .. _plugin-cutter: https://github.com/aiidateam/aiida-plugin-cutter
 .. _aiida-diff: https://github.com/aiidateam/aiida-diff
 .. _pytest: https://pytest.org