Testing plugin against latest AiiDA

The following will show how you can setup a nightly build that will run the test suite of your plugin against the latest develop branch of aiida-core. This is not only useful for yourself, but also for us as AiiDA developers. If a build fails, you can notify us immediately if it is due to us (accidentally) breaking the interface. If this is the case, we can correct the mistake before releasing the code.

Pre-requisites

There are many ways of accomplishing what is described above, but in this example the following is required:

1. Your plugin is hosted on Github
2. You have a Slack project

How does it work

The idea is to add a Github Action to your repository that will run every day at midnight on the default branch, which is usually master or develop. You can check the current default or change it by clicking on Settings and then Branches. If your repository is hosted on Github, doing so is easy, there is just a single file you have to add to your repository and the action will start to run. There is nothing you have to activate on Github itself. After that, each day at midnight your test suite will run against the latest develop branch of aiida-core. If it fails, a message will be sent to the Slack channel that you can configure. This way you can check the failing build and see if it is due to a breaking change in aiida-core, in which case it would be great if you can open an issue on aiida-core.

Step-by-step instructions

1. Create the folder .github/workflows in your repository
2. Create a file called develop.yml in that directory and add the following content:

name: Nightly Build

on:
    schedule:
    -   cron: '0 0 * * *'  # Run every day at midnight

jobs:

    tests:

        if: github.repository == 'aiidateam/aiida-quantumespresso'  # Prevent running the builds on forks as well

        runs-on: ubuntu-latest

        strategy:
            matrix:
                python-version: [3.6, 3.7, 3.8]

        services:
            postgres:
                image: postgres:10
                env:
                    POSTGRES_DB: test_${{ matrix.backend }}
                    POSTGRES_PASSWORD: ''
                    POSTGRES_HOST_AUTH_METHOD: trust
                ports:
                -    5432:5432
            rabbitmq:
                image: rabbitmq:latest
                ports:
                -   5672:5672

        steps:
        -   uses: actions/checkout@v2

        -   name: Cache python dependencies
            id: cache-pip
            uses: actions/cache@v1
            with:
                path: ~/.cache/pip
                key: pip-${{ matrix.python-version }}-tests-${{ hashFiles('**/setup.json') }}
                restore-keys:
                    pip-${{ matrix.python-version }}-tests

        -   name: Set up Python ${{ matrix.python-version }}
            uses: actions/setup-python@v2
            with:
                python-version: ${{ matrix.python-version }}

        -   name: Install system dependencies
            run: |
                sudo apt update
                sudo apt install postgresql-10

        -   name: Install python dependencies
            continue-on-error: true
            id: install
            run: |
                pip install --upgrade setuptools
                pip install -e .[tests]
                pip install git+https://github.com/aiidateam/aiida-core@develop
                reentry scan

        -   name: Run pytest
            continue-on-error: true
            id: tests
            run:
                pytest -sv tests

        -   name: Slack notification
            if: steps.install.outcome == 'Failure' || steps.tests.outcome == 'Failure'
            uses: rtCamp/action-slack-notify@master
            env:
                SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
                SLACK_ICON: https://www.materialscloud.org/discover/images/0ba0a17d.aiida-logo-128.png
                SLACK_USERNAME: aiida-quantumespresso
                SLACK_CHANNEL: dev-aiida-qe
                SLACK_COLOR: b60205
                SLACK_TITLE: "Nightly build against `aiida-core/develop` failed"
                SLACK_MESSAGE: "The tests fail with the current version of `aiida-core/develop`."

3. Replace the string aiidateam/aiida-quantumespresso with the name of your organization/user and repository.
4. Make sure to update the line pytest -sv tests with the correct way of invoking your test suite if you do not run pytest or the tests are not stored in the tests folder.
5. (Optional) Replace the value for SLACK_USERNAME with that of your plugin.
6. Create a dedicated channel on your Slack project (can also use existing channel).
7. Add an incoming webhook token for your Slack project.
8. Follow the instructions here to generate a Github Secret. When asked for the name enter SLACK_WEBHOOK and for the value, enter the URL that was generated by Slack in the previous step.
9. Create a commit to add the new workflow file and merge it to your default branch.

That is it. The next time around midnight, your build should automatically start to run. You can check it out by clicking on the Actions tab on your repository page in Github.