Establish Python project template used throughout repos.

[yakutovicha]

Currently, there are multiple ways a Python project can be organized.

Here are the choices

• Python management tool: setuptools, flit, hatch, poetry. Current choice: [flit for pure Python packages, setuptools for everything else].
• Linters: flake8, pylint. Current choice: [flake8]. Additional flake8-plugins:
  • flake8-bugbear for detecting bugs and potential design problems
  • flake8-builtins for verifying that builtins are not used as variable names
  • flake8-comprehensions for writing better and consistent comprehensions
  • flake8-debugger for checking that there are no forgotten breakpoints
  • flake8-eradicate to avoid “dead” or commented code
  • flake8-logging-format for consistent logging
  • pep8-naming for verifying that PEP8 naming conventions are followed
  • tryceratops to avoid exception anti-patterns
• Type checking: mypy. Current choice: [mypy]
  • types-click-spinner
  • types-requests, types-tabulate, types-toml
• Auto-formatters:
  • isort for sorting imports
  • autoflake for removing unused imports
  • black for formatting the rest of the code.
• Python auto-update: pyupgrade tool to automatically upgrade syntax for newer versions of the language.
• Release automation: bumpver. Current choice: [bumpver]
• Testing: pytest. Current choice: pytest.
• Documentation: to complete

Currently, I am working on a Python cookie-cutter repository to establish those. The issue is to be closed when all choices are made and the template repository reflects them.

The cookie-cutter repository will be largely based on https://github.com/woltapp/wolt-python-package-cookiecutter. I will also create an accompanying template similar to https://github.com/simonw/python-lib-template-repository. The idea is that the latter takes the cookie-cutter one, fills in the values and annihilates himself (see this file for the details).

[yakutovicha]

I believe a lot of inspiration can be taken from this blog post. Not only do they have a nice python-project template, but also they are able to perform bulk actions on already created repositories

[yakutovicha]

setuptools vs flint.

Flit is very simple to set up, it has built-in git integration (no need for the MANIFEST file anymore), and it reads version/description directly from the package __init__.py. So one doesn't need to set the version in multiple places.

So, it seems that for the pure python packages - flit is a better choice.

[yakutovicha]

.pre-commit-config.yaml

I would propose using the following configuration:

---
ci:
    autoupdat_schedule: quarterly

repos:
    - repo: https://github.com/pre-commit/pre-commit-hooks
      rev: v4.4.0
      hooks:
          - id: end-of-file-fixer
            exclude: ^.*data/
          - id: trailing-whitespace
            exclude: miscellaneous/structures/SiO2.xyz
          - id: check-yaml
          - id: check-added-large-files
    - repo: https://github.com/pycqa/isort
      rev: 5.12.0
      hooks:
          - id: isort
            args: [--profile, black, --filter-files]
    - repo: https://github.com/PyCQA/autoflake
      rev: v2.1.1
      hooks:
          - id: autoflake
    - repo: https://github.com/asottile/pyupgrade
      rev: v3.4.0
      hooks:
          - id: pyupgrade
            args: [--py38-plus]
    - repo: https://github.com/psf/black
      rev: 23.3.0
      hooks:
          - id: black
            language_version: python3 # Should be a command that runs python3.6+
    - repo: https://github.com/PyCQA/flake8
      rev: 6.0.0
      hooks:
          - id: flake8
            args: [--count, --show-source, --statistics]
            additional_dependencies:
                - flake8-bugbear
                - flake8-builtins
                - flake8-comprehensions
                - flake8-debugger
                - flake8-logging-format
                - pep8-naming
                - pyflakes
                - tryceratops
    - repo: https://github.com/pre-commit/mirrors-mypy
      rev: v1.3.0
      hooks:
          - id: mypy
            additional_dependencies:
                - types-click-spinner
                - types-requests
                - types-tabulate
                - types-toml
    - repo: https://github.com/jumanjihouse/pre-commit-hook-yamlfmt
      rev: 0.2.3
      hooks:
          - id: yamlfmt

if the repo contains notebooks, it makes sense to add the following

    - repo: https://github.com/kynan/nbstripout
      rev: 0.6.1
      hooks:
          - id: nbstripout

    - repo: https://github.com/nbQA-dev/nbQA
      rev: 1.7.0
      hooks:
          - id: nbqa-pyupgrade
            args: [--py38-plus]
          - id: nbqa-isort
            args: [--profile=black]
          - id: nbqa-black

if the repo contains setup.cfg, add the following

    - repo: https://github.com/asottile/setup-cfg-fmt
      rev: v2.2.0
      hooks:
          - id: setup-cfg-fmt

[yakutovicha]

Flake8 configuration file.

It would be great, of course, to make the config below a part of the pyproject.toml. Unfortunately, this is not possible. Nevertheless, this is the content of .flake8 (needed mostly due to the conflicts with black):

[flake8]
extend-ignore =
    E501
    W503
    E203

[yakutovicha]

There is also black for notebooks. Here is a nice way to configure it:

aiidalab/aiidalab-qe#396