Merge branch 'develop' into jandom/feat/add-snapshot-tests-with-syrup

.github/workflows/gh-ci-tests.yaml

@@ -12,7 +12,7 @@ on:
 
 concurrency:
   # Probably overly cautious group naming.
-  # Commits to develop/master will cancel each other, but PRs will only cancel
+  # Commits to develop will cancel each other, but PRs will only cancel
   # commits within the same PR
   group: "${{ github.ref }}-${{ github.head_ref }}-${{ github.workflow }}"
   cancel-in-progress: true

.github/workflows/gh-ci.yaml

@@ -3,20 +3,18 @@ on:
   push:
     branches:
       - develop
-      - master
     tags:
       - "release-*"
   pull_request:
     branches:
       - develop
-      - master
   schedule:
     # 3 am Tuesdays and Fridays
     - cron: "0 3 * * 2,5"
 
 concurrency:
   # Probably overly cautious group naming.
-  # Commits to develop/master will cancel each other, but PRs will only cancel
+  # Commits to develop will cancel each other, but PRs will only cancel
   # commits within the same PR
   group: "${{ github.ref }}-${{ github.head_ref }}-${{ github.workflow }}"
   cancel-in-progress: true

.github/workflows/pre-commit.yaml

@@ -10,7 +10,7 @@ on:
 
 concurrency:
   # Probably overly cautious group naming.
-  # Commits to develop/master will cancel each other, but PRs will only cancel
+  # Commits to develop will cancel each other, but PRs will only cancel
   # commits within the same PR
   group: "${{ github.ref }}-${{ github.head_ref }}-${{ github.workflow }}"
   cancel-in-progress: true

README.rst

@@ -15,7 +15,7 @@ This repository contains the `user guide`_ for MDAnalysis. The user guide origin
 .. _`user guide`: https://www.mdanalysis.org/UserGuide/index.html
 .. _`many popular formats`: https://www.mdanalysis.org/UserGuide/formats/index.html
 .. _MDAnalysis: https://www.mdanalysis.org
-.. _LICENSE: https://github.com/MDAnalysis/mdanalysis/blob/master/LICENSE
+.. _LICENSE: https://github.com/MDAnalysis/mdanalysis/blob/develop/LICENSE
 .. _`project for Google Season of Docs 2019-2020`: https://developers.google.com/season-of-docs/docs/participants/project-mdanalysis
 
 .. |usergroup| image:: https://img.shields.io/badge/Google%20Group-Users-lightgrey.svg

doc/source/contributing_code.rst

@@ -270,9 +270,6 @@ At this point you should be able to import MDAnalysis from your locally built ve
         >>> mda.__version__
         '2.6.0-dev0'
 
-.. note::
-    If your version number does not end in :code:`-dev0`, you may be on the ``master`` branch. In your ``mdanalysis/`` directory, switch to the ``develop`` branch:
-    :code:`git checkout develop`.
 
 With pip and virtualenv
 -----------------------
@@ -343,25 +340,16 @@ At this point you should be able to import MDAnalysis from your locally built ve
         >>> mda.__version__
         '2.6.0-dev0'
 
-.. note::
-    If your version number does not end in :code:`-dev0`, you may be on the ``master`` branch. In your ``mdanalysis/`` directory, switch to the ``develop`` branch:
-    :code:`git checkout develop`.
-
 
 .. _branches-in-mdanalysis:
 
 ----------------------
 Branches in MDAnalysis
 ----------------------
 
-There are two important branches in MDAnalysis:
-
-    - ``master``: for production-ready code
-    - ``develop``: for development code
+The most important branch of MDAnalysis is the ``develop`` branch, to which all development code for the next release is pushed.
 
-The ``master`` branch is only for stable, production-ready code. Development code should *never* be committed to this branch. Typically, code is only committed by the release manager, when a release is ready.
-
-The ``develop`` branch can be considered an "integration" branch for including your code into the next release. Only working, tested code should be committed to this branch. Code contributions ("features") should branch off ``develop`` rather than ``master``.
+The ``develop`` branch can be considered an "integration" branch for including your code into the next release. Only working, tested code should be committed to this branch. All code contributions ("features") should branch off ``develop``. At each release, a snapshot of the ``develop`` branch is taken, packaged and uploaded to PyPi and conda-forge.
 
 
 .. _create-code-branch:
@@ -386,10 +374,16 @@ There are several special branch names that you should not use for your feature
 
     - ``master``
     - ``develop``
-    - ``release-*``
+    - ``package-*``
+    - ``gh-pages``
+
+
+``package`` branches are used to :ref:`prepare a new production release <preparing-release>` and should be handled by the release manager only.
+
+``master`` is the old stable code branch and is kept protected for historical reasons.
 
+``gh-pages`` is where built documentation to be uploaded to github pages is held.
 
-``release`` branches are used to :ref:`prepare a new production release <preparing-release>` and should be handled by the release manager only.
 
 .. _writing-new-code:
 
@@ -673,7 +667,7 @@ in the Scientific Python community. They are nice to read as normal text and are
 
 This standard specifies the format of
 the different sections of the docstring. See `this document
-<https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt>`_
+<https://numpydoc.readthedocs.io/en/latest/format.html#docstring-standard>`_
 for a detailed explanation, or look at some of the existing functions to
 extend it in a similar manner.
 

doc/source/contributing_docs.rst

@@ -42,7 +42,7 @@ Here is an overview of the development workflow for the user guide, as expanded
 
     #. :ref:`Fork the MDAnalysis repository <forking-user-guide>` from the mdanalysis account into your own account
     #. :ref:`Set up an isolated virtual environment <create-virtual-environment-user-guide>` for your documentation
-    #. :ref:`Create a new branch off the master branch <create-code-branch>`
+    #. :ref:`Create a new branch off the develop branch <create-code-branch>`
     #. Add your new documentation.
     #. :ref:`Build and view the documentation <build-user-guide>`.
     #. :ref:`Commit and push your changes, and open a pull request <add-changes-user-guide>`.

doc/source/examples/README.rst

@@ -38,4 +38,4 @@ each notebook on `Binder`_.
    :target: https://mybinder.org/v2/gh/MDAnalysis/binder-env-userguide/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FMDAnalysis%252FUserGuide%26urlpath%3Dtree%252FUserGuide%252Fdoc%252Fsource%252Fexamples%26branch%3Dmaster
 
 .. _Binder: https://mybinder.org/v2/gh/MDAnalysis/binder-env-userguide/main?urlpath=git-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FMDAnalysis%252FUserGuide%26urlpath%3Dtree%252FUserGuide%252Fdoc%252Fsource%252Fexamples%26branch%3Dmaster
-.. _GitHub: https://github.com/MDAnalysis/UserGuide/tree/master/doc/source/examples
+.. _GitHub: https://github.com/MDAnalysis/UserGuide/tree/develop/doc/source/examples

doc/source/preparing_releases_and_hotfixes.rst

@@ -5,11 +5,10 @@
 Preparing a release
 ===================
 
-Rules for a release branch:
+Rules for release branches:
 
-    - May branch from: ``develop``
-    - Must be merged into: ``master`` (and ``develop`` if needed)
-    - Naming convention: ``release-*`` where ``*`` is a version number
+    - Branch from ``develop``
+    - Naming convention: ``package-*`` where ``*`` is a version number
 
 Release policy and release numbering
 ====================================
@@ -44,27 +43,47 @@ so that people using the :ref:`develop branch <branches-in-mdanalysis>` from the
 Typical workflow for preparing a release
 ========================================
 
-#. Declare feature freeze on ``develop`` via the `developer mailing list`_
+Summary of tasks
+----------------
 
-#. Create a release branch from ``develop``::
+  * Declare a `feature freeze` on `develop` via discord and/or the developer mailing list
+  * Finalize the ``CHANGELOG`` file with the date of release
+  * Increment the version across ``MDAnalysis`` and ``MDAnalysisTests`` (4 places)
+  * `Merge` changes into `develop`
+  * Create a tag from `develop` named `release-<version_number>`
+  * Create a `package-<version_number>` branch from `develop`
+  * Add rebuilt `C / C++` files to `package-<version_number>`
+  * Create a tag from `package-<version_number>`
+  * Check automated testing of source distribution and wheel generation and upload
+  * Create a new release from newly created tag
+  * Check that deployment actions have adequately pushed dist and wheels to PyPi
+  * Manually upload Cirrus CI wheels (temporary)
+  * Update `conda-forge` packages
+  * Create a blog post outlining the release
+  * Increment develop branch files ready for the next version
+  * Clean up old dev docs builds
 
-    git checkout -b release-0.7.6 develop
 
-#. Finalise the ``CHANGELOG`` with the release number and date. Summarize important changes and add all authors that contributed to this release.
+Getting the develop branch ready for a release
+----------------------------------------------
 
-#. Make sure the version number is right. The following files need to be updated: ``maintainer/conda/MDAnalysis/meta.yaml``, ``package/MDAnalysis/version.py``, ``package/setup.py``, ``testsuite/MDAnalysisTests/__init__.py``, and ``testsuite/setup.py``.
+#. Declare feature freeze on ``develop`` via `discord` and the `developer mailing list`_
 
-#. Check that the documentation is up-to-date and tests pass.
+#. Create a pre-release feature branch from ``develop``
 
-#. Commit the finalized state::
+#. Finalise the ``CHANGELOG`` with the release number and date. Ensure that the ``CHANGELOG`` summarizes important changes and includes all authors that contributed to this release.
 
-    git commit -m "release 0.7.6 ready"
+#. Make sure the version number matches the release version. The following files need to be updated: ``package/MDAnalysis/version.py``, ``package/setup.py``, ``testsuite/MDAnalysisTests/__init__.py``, and ``testsuite/setup.py``.
 
 #. Create a pull request against ``develop`` from this branch.
 
+
+Packaging the release
+---------------------
+
 #. Create a new tag from ``develop`` named ``release-<version_number>`` where ``<version_number>`` is the release version number (this tag contains a snapshot of the Python source files as they were when the release was created):
 
-    .. code-block: bash
+    .. code-block:: bash
 
         git tag -m "release 0.7.5 of MDAnalysis and MDAnalysisTests" release-0.7.5
         git push --tags origin
@@ -77,7 +96,7 @@ Typical workflow for preparing a release
 
         # MDAnalysis
         cd package/
-        python setup.py sdist
+        pipx run build --sdist
 
 #. Once committed, create a new tag based on ``package-<version_number>`` (this tag will contain a record of all the files as they were deployed to users for that version):
 
@@ -86,7 +105,7 @@ Typical workflow for preparing a release
         git tag -m "package 0.7.5 of MDAnalysis and MDAnalysisTests" package-0.7.5
         git push --tags origin
 
-#. Upon creation of the new ``package-*`` tag, the `developer mailing list`_, will be triggered to create source/wheels, upload them to testpypi, and re-download them and run tests.
+#. Upon creation of the new ``package-*`` tag, the `deploy github action`_ workflow will be automatically triggered to create source/wheels, upload them to testpypi, re-download them and run tests.
 
 #. If all the tests come back green, you are good to go for a full release.
 
@@ -102,33 +121,83 @@ Typical workflow for preparing a release
 
             #. Add the new changes to ``develop`` and restart the release process.
 
-            #. Upon getting to the ``package-*`` tag creation, create a test tag with a different version number (bump by a minor release or add a ``-beta`` modifier), so as to trigger the release and tests against testpypi.
+            #. If the code had successfully uploaded to testpypi and failed later, you will need to create a test ``package-*`` tag which contains a different release number of in the source code (bumpy by a minor release or add a ``-beta`` modifier). Note: if the code had not successfully uploaded you can just continue the release process as normal.
+
+            #. If CI comes back green then delete the test tag, and create a normal ``package-*`` tag with the correct version number.
+
+            #. The github action will fail, but this is ok since we tested it with the test tag above.
+
+
+Completing the release
+----------------------
+
+If everything works, you can now complete the release by:
+
+#. Creating a release on GitHub based on the newly created ``package-<version_number>`` tag.
+
+#. Make sure you include relevant release notes, including any known issues and highlights for the release.
+
+#. Once published, the `deploy github action`_ will be triggered which will upload the source distributions and wheels to PyPI.
+
+    #. If the `deploy github action`_ fails and no files have been uploaded, then restart the action.
+
+    #. If the action fails and some files have been uploaded, then you will not be able to re-upload to PyPI. At this point you will need to yank the release from PyPI and create a new minor version and re-deploy it.
+
+
+Manually upload Cirrus CI wheels (temporary)
+--------------------------------------------
+
+
+.. todo:: actually add some examples & links of how to do this.`
+
+
+Unfortunately the deployment of Cirrus CI generated wheels (for `osx-arm64` and `linux-aarch64`) does not get properly triggered by a release.
+
+#. Go to the ``package-*`` tag triggered Cirrus CI run results and download the generated wheels.
+
+#. Upload them to PyPi using ``twine``.
+
+
+Update `conda-forge` packages
+-----------------------------
+
+On push to PyPI, the conda-forge bot should automatically pick up the presense of a new version and create a pull request on the `MDAnalysis feedstock`_ and the `MDAnalysisTests feedstock`_. You will need to merge the MDAnalysis feedstock followed by the MDAnalysisTests feedstock in order for the new package to appear on conda-forge.
+
+To do this you will need to:
+
+#. Update the ``meta.yaml`` files as necessary, especially bumping up the python and dependency minimum versions as necessary.
+
+#. If NumPy pins differ from those conda-forge uses, you will need to update the ``conda_build_config.yaml`` accordingly.
+
+#. Ask the conda-forge bot to re-render, check that CI returns green, approve and merge the pull request.
+
 
-            #. Delete the test tag, create a normal ``package-<version_number>`` tag with the correct version number.
+Create a blog post outlining the release
+----------------------------------------
 
-            #. The githuba action will fail, but this is ok since we tested it with the test tag above.
+Create a blog post outlining the release notes and publicize it on the mailing list / discord / twitter/ etc...!
 
-#. If everything works, you can now complete the release by createing a release on Github based on the newly created ``package-<version_number`` tag.
 
-#. This will re-trigger the `deploy github action`_ and upload the source distributions / wheels to PyPI.
+Increment develop branch files ready for the next version
+---------------------------------------------------------
 
-    #. If the action fails as this point and no files have been deployed, then restart the action.
+Once the release is completed you can go ahead and update the ``develop`` branch so that it is ready for the next round of development.
 
-    #. If the action fails and some files have been deployed, then you will not be able to re-upload to PyPI. At this point you will need to yank the release from PyPI and create a new minor version and re-deploy it.
+#. Update the 4 version file locations with the ``-dev0`` appended version of the next release.
 
-#. Update the release on conda-forge
+#. Update the CHANGELOG with a new entry for the next release.
 
-    #. On push to PyPI, the conda-forge bot should automatically pick up the presense of a new version and create a pull request on the `MDAnalysis feedstock`_
+#. Once these changes are merged into the ``develop`` branch, message the developers on discord and the mailing list letting them know that the feature freeze is over.
 
-    #. Update the ``meta.yaml`` file as necessary, especially bumping up the python and dependency minimum versions as necessary.
 
-    #. Ask the conda-forge admin to re-render, check that CI returns green, approve and merge the pull request.
+Clean up old developer builds of the documentation
+--------------------------------------------------
 
-#. Documentation will be built automatically and versioned. Check that these have been created appropriately on the `stable branch of the docs page`_.
+Whilst new docs are automatically deployed on a release, old developer builds (appended with ``-dev``) are not automatically cleaned up. To avoid causing large amounts of files being uploaded to GitHub Pages, we need to delete these old developer builds manually. To do this switch to the ``gh-pages`` branch, delete these old files, and push the change directly.
 
-#. Create a blog post outlining the release notes and publicize it on the mailing list / discord / twitter/ etc...!
 
 .. _`developer mailing list`: https://groups.google.com/forum/#!forum/mdnalysis-devel
 .. _`deploy github action`: https://github.com/MDAnalysis/mdanalysis/tree/develop/.github/workflows/deploy.yaml
 .. _`MDAnalysis feedstock`: https://github.com/conda-forge/mdanalysis-feedstock
+.. _`MDAnalysisTests feedstock`: https://github.com/conda-forge/mdanalysistests-feedstock
 .. _`stable branch of the docs page`: https://docs.mdanalysis.org/stable