Streamline continuous integration: reduce number of wheels, concurren…

…cy (#182)

* Only generate full wheels on releases, pull requests or main branch

* for regular branch commits only build for Python 3.12 on Linux/Mac/Windows. (i.e. non-main, non-PR and non-releases)

* concurrency: ensure that new PR pushes cancel current workflows (https://learn.scientific-python.org/development/guides/gha-basic/#cancel-existing-runs)

* cleanup workflow files: move unused stuff to DEV_NOTES.md

* docs should also be rebuilt upon changes to NEWS.md

* group updates for dependabot

.github/dependabot.yml

@@ -5,6 +5,10 @@ updates:
   directory: "/"
   schedule:
     interval: weekly
+  groups:
+    actions:
+      patterns:
+        - "*"
 
 # for pinned Python packages
 - package-ecosystem: "pip"

.github/workflows/build_wheels.yml

@@ -1,5 +1,9 @@
 name: Build PyPop
 
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
+
 on:
   workflow_dispatch:
   pull_request:
@@ -92,11 +96,6 @@ jobs:
           git ls-files -z|git check-attr --stdin -z export-ignore |sed -zne 'x;n;n;s/^set$//;t print;b;:print;x;p'|xargs --null rm
           # FIXME: need to also remove orphaned empty directories resulting from above
           find . -empty -type d -delete
-          # git tag -d $GITHUB_REF_NAME
-          # git push --follow-tags origin :$GITHUB_REF
-          # git tag $GITHUB_REF_NAME
-          # git push origin $GITHUB_REF
-          # git ls-remote origin $GITHUB_REF
       - name: Create a draft snapshot of your repository contents as a new
               version in test collection on Zenodo using metadata
               from repository file .zenodo.json
@@ -114,43 +113,6 @@ jobs:
           upsert-doi: true
           upsert-location: doi
           verbose: true
-      #
-      # FIXME: disabled the rest of these steps (already done by zenodraft)
-      #
-      - name: Get existing release
-        if: false
-        id: get_existing_release
-        uses: cardinalby/git-get-release-action@v1
-        env:
-          GITHUB_TOKEN: ${{ github.token }}
-        with:
-          releaseId: ${{ github.event.release.id }}
-      - name: Delete old release
-        if: false
-        uses: liudonghua123/delete-release-action@v1
-        env:
-          GITHUB_TOKEN: ${{ github.token }}
-        with:
-          release_id: ${{ github.event.release.id }}
-      - name: Recreate release with new tag
-        if: false
-        id: recreate_release
-        uses: joutvhu/create-release@v1
-        with:
-          tag_name: ${{ github.event.release.tag_name }}
-          name: ${{ steps.get_existing_release.outputs.name }}
-          body: ${{ steps.get_existing_release.outputs.body }}
-          # FIXME: have to set these both to false, because not copied
-          # from the original release properly
-          # draft: ${{ steps.get_existing_release.outputs.draft }}
-          # prerelease: ${{ steps.get_existing_release.outputs.draft }}
-          draft: false
-          prerelease: false
-          target_commitish: ${{ steps.get_existing_release.outputs.target_commitish }}
-          on_release_exists: update
-        env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          # GITHUB_TOKEN: ${{ secrets.PYPOP_RELEASE_TOKEN }}
 
   generate-wheels-matrix:
     # Create a matrix of all architectures & versions to build.
@@ -167,18 +129,27 @@ jobs:
         run: pipx install cibuildwheel==2.16.2
       - id: set-matrix
         run: |
-          MATRIX=$(
-            {
+          echo "CI_ONLY" $CI_ONLY
+          if [ "$CI_ONLY" == "true" ]; then
+            MATRIX="[{'only':'cp312-manylinux_x86_64','os':'ubuntu-20.04'},{'only':'cp312-win_amd64','os':'windows-2019'}, {'only':'cp312-macosx_x86_64','os':'macos-11'}, {'only':'cp312-macosx_arm64','os':'macos-11'}]"
+          else
+            MATRIX=$(
+              {
               cibuildwheel --print-build-identifiers --platform linux \
               | jq -nRc '{"only": inputs, "os": "ubuntu-20.04"}' \
               && cibuildwheel --print-build-identifiers --platform macos \
               | jq -nRc '{"only": inputs, "os": "macos-11"}' \
               && cibuildwheel --print-build-identifiers --platform windows \
               | jq -nRc '{"only": inputs, "os": "windows-2019"}'
             } | jq -sc
-          )
+            )
+          fi
           echo "include=$MATRIX" >> $GITHUB_OUTPUT
-          echo "debugging:" $MATRIX
+          echo "platform matrix:" $MATRIX
+        env:
+           # only generate all wheels on releases, pull requests or commits to the main branch
+           # otherwise just generate a test wheel Python 3.12 for 4 platform
+           CI_ONLY: ${{ github.event_name != 'release' && github.event_name != 'pull_request' && github.ref != 'main' }}
 
   build_wheels:
     name: Build wheels on ${{ matrix.only }}

.github/workflows/documentation.yaml

@@ -1,8 +1,15 @@
 name: pypop.org website and docs generation
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+  cancel-in-progress: true
+
 on:
+  workflow_dispatch:
   push:
     paths:
       - '*.rst'
+      - 'NEWS.md'
       - 'LICENSE'
       - 'website/**'
       - 'tests/data/doc-examples/**'
@@ -13,16 +20,7 @@ on:
   pull_request:
     paths:
       - '*.rst'
-      - 'LICENSE'
-      - 'website/**'
-      - 'tests/data/doc-examples/**'
-      - 'src/PyPop/CommandLineInterface.py'
-      - '.github/workflows/documentation.yaml'
-      - '!website/README.md'
-      - '!website/reference/**'
-  workflow_dispatch:
-    paths:
-      - '*.rst'
+      - 'NEWS.md'
       - 'LICENSE'
       - 'website/**'
       - 'tests/data/doc-examples/**'

DEV_NOTES.md

@@ -6,86 +6,151 @@
   `haplo.stats` R package `haplo-stats` for haplotype
   estimation. [Implementation in alpha-phase - still working on this].
 
+## Build-time notes
 
-## External dependencies
+### External dependencies
 
 * ```swig``` (Simple Wrapper Interface Generator) (build-time only)
-* ```gsl``` (GNU Scientific Library)
+* ```gsl``` (GNU Scientific Library) (build-time only)
 * ```Numpy``` (Numpy)
 * ```lxml``` (Python bindings)
 * ```pytest``` (Python test framework)
 
-## Installing ```swig``` on certain Ubuntu releases (obsoleted by newer Ubuntu releases)
+### SWIG notes
 
-There is a bug in versions swig 3.0.6 to 3.0.10 that prevents swig on ```xenial``` (which is version 3.0.8 of swig) working.  You will need to install the lastest version from source.
+* Note that within ".i" wrappers, need to include function prototypes
+and SWIG wrappers, so functions are duplicated, see this
+[StackOverflow
+post](https://stackoverflow.com/questions/66995429/cant-run-swig-tutorial-for-python)
 
-1. Get swig dependency: 
+### macports.org
 
-```sudo apt install libpcre3-dev```
+* To install macports via the command-line you can run the following (substituting the current link):
 
-2. Visit [swig.org](swig.org) to get download link
+```
+curl -L 'https://github.com/macports/macports-base/releases/download/v2.4.1/MacPorts-2.4.1-10.12-Sierra.pkg' > MacPorts-2.4.1-10.12-Sierra.pkg
+sudo installer -pkg MacPorts-2.4.1-10.12-Sierra.pkg  -target /
+```
 
-3. Do the installation:
+## GitHub notes
 
-       tar zxvf ~/swig-3.0.12.tar.gz
-       cd swig-3.0.12
-       ./configure
-       make
-       sudo make install
+Unused stanzas in `build_wheels.yml` workflow to manually regenerate a
+GitHub release based on re-tagging (e.g. after a Zenodo deposition).
+They have been disabled for the time being because they are done
+internally by the `zenodraft` action.
+
+This first part is needed just after `find -empty type d -delete` in
+the `run` part of the "Pubsh changes back to repo files"
+`publish_zenodo` job:
 
+```
+          git tag -d $GITHUB_REF_NAME
+          git push --follow-tags origin :$GITHUB_REF
+          git tag $GITHUB_REF_NAME
+          git push origin $GITHUB_REF
+          git ls-remote origin $GITHUB_REF
+```
+
+This second part would be the last step in the same `publish_zenodo`
+job:
+
+```
+      #
+      # FIXME: disabled the rest of these steps (already done by zenodraft)
+      #
+      - name: Get existing release
+        if: false
+        id: get_existing_release
+        uses: cardinalby/git-get-release-action@v1
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+        with:
+          releaseId: ${{ github.event.release.id }}
+      - name: Delete old release
+        if: false
+        uses: liudonghua123/delete-release-action@v1
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+        with:
+          release_id: ${{ github.event.release.id }}
+      - name: Recreate release with new tag
+        if: false
+        id: recreate_release
+        uses: joutvhu/create-release@v1
+        with:
+          tag_name: ${{ github.event.release.tag_name }}
+          name: ${{ steps.get_existing_release.outputs.name }}
+          body: ${{ steps.get_existing_release.outputs.body }}
+          # FIXME: have to set these both to false, because not copied
+          # from the original release properly
+          # draft: ${{ steps.get_existing_release.outputs.draft }}
+          # prerelease: ${{ steps.get_existing_release.outputs.draft }}
+          draft: false
+          prerelease: false
+          target_commitish: ${{ steps.get_existing_release.outputs.target_commitish }}
+          on_release_exists: update
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          # GITHUB_TOKEN: ${{ secrets.PYPOP_RELEASE_TOKEN }}
+```
 
 ## Design notes
 
-* 'Main' is the primary interface to the PyPop modules.  Given a
+(These should eventually be migrated back in the source code, so that,
+if and when we generate API docs, they will appear there.  They would
+need updating as part of that migration to make sure they are still
+accurate.)
+
+* `Main` is the primary interface to the PyPop modules.  Given a
 ConfigParser instance, which can be (1) created from a filename passed
 from command-line argument or (2) from values populated by the GUI
 (currently selected from an .ini file, but can ultimately be set
 directly from the GUI or values based from a form to a web server or
 the) it then runs the specified modules (outlined below).
 
-* 'GUIApp' is the graphical front-end to PyPop which uses the
+* `GUIApp` is the graphical front-end to PyPop which uses the
 "wxPython":http://www.wxpython.org GUI toolkit.  wxPython is a set of
 Python bindings to "wxWindows":http://www.wxwindows.org, which is an
 open-source cross-platform GUI widget toolkit which has a native look
 under GNU/Linux (GTK), Windows (MFC) and MacOS X (Aqua).  [as of 2023,
 this was removed]
 
-* 'ParseFile' is a base class which has most of the common functionality
+* `ParseFile` is a base class which has most of the common functionality
 for reading files.
 
-* 'ParseGenotypeFile' is a subclass of 'ParseFile' that deals with
+* `ParseGenotypeFile` is a subclass of `ParseFile` that deals with
 files that consist specifically of data with individual genotyped for
 one or more loci.
 
-* 'ParseAlleleCount' is another subclass of 'ParseFile' that deals
+* `ParseAlleleCount` is another subclass of `ParseFile` that deals
 with files consisting of allele counts across a whole population.
 
-* 'HardyWeinberg' is a class that calculates Hardy-Weinberg
+* `HardyWeinberg` is a class that calculates Hardy-Weinberg
 statistics given genotype data for a single locus.
 
-* 'HardyWeinbergGuoThompson' a subclass of 'HardyWeinberg' that uses the
+* `HardyWeinbergGuoThompson` a subclass of `HardyWeinberg` that uses the
 Guo & Thompson algorithm for calculating statistics.
 
-* 'HardyWeinbergGuoThompsonArlequin' a subclass of 'HardyWeinberg'
+* `HardyWeinbergGuoThompsonArlequin` a subclass of `HardyWeinberg`
 that uses the Arlequin implementation of the Guo & Thompson algorithm
 for calculating statistics.
 
-* 'Haplo' is an abstract base class for estimating haplotypes given
+* `Haplo` is an abstract base class for estimating haplotypes given
 genotype data.
 
-- 'HaploArlequin' is a subclass of 'Haplo' that uses Arlequin for
+- `HaploArlequin` is a subclass of `Haplo` that uses Arlequin for
 estimation of haplotypes (obsolete).
 
-* 'Emhaplofreq' is a subclass of 'Haplo' that uses 'emhaplofreq' (Rich
-Single's program) for the estimation of haplotypes and linkage
+* `Emhaplofreq` is a subclass of `Haplo` that uses `emhaplofreq` (Rich
+Single`s program) for the estimation of haplotypes and linkage
 disequilibrium values.
 
-* 'ArlequinWrapper' the underlying class that `wraps' the
+* `ArlequinWrapper` the underlying class that "wraps" the
 functionality of the "Arlequin":http://lgb.unige.ch/arlequin/ program
-(obsolete: this class, in turn, supplies 'HaploArlequin' with required
+(obsolete: this class, in turn, supplies `HaploArlequin` with required
 information).
 
-* 'Homozygosity' Calculates homozygosity statistics for a given locus,
+* `Homozygosity` Calculates homozygosity statistics for a given locus,
 calculates the observed homozygosity and returns the approximate
 expected homozygosity statistics taken from previous simulation runs.
 
@@ -96,23 +161,38 @@ headers for the samples (sample metadata) followed by the sample data
 itself (either individuals in the genotyped case, or alleles in the
 allele count case).
 
-## SWIG notes
+## Obsolete notes
 
-* Note that within ".i" wrappers, need to include function prototypes
-and SWIG wrappers, so functions are duplicated, see this
-[StackOverflow
-post](https://stackoverflow.com/questions/66995429/cant-run-swig-tutorial-for-python)
+These are either obsoleted by new versions of dependencies or
+platforms, or no longer work, and need to be updated. Keeping around
+in case of either old platforms or if there is interest in reviving
+the feature(s) in question.
 
-## macports.org
+### Installing ```swig``` on certain Ubuntu releases
 
-* To install macports via the command-line you can run the following (substituting the current link):
+(obsoleted by newer Ubuntu releases)
 
-```
-curl -L 'https://github.com/macports/macports-base/releases/download/v2.4.1/MacPorts-2.4.1-10.12-Sierra.pkg' > MacPorts-2.4.1-10.12-Sierra.pkg
-sudo installer -pkg MacPorts-2.4.1-10.12-Sierra.pkg  -target /
-```
+There is a bug in versions swig 3.0.6 to 3.0.10 that prevents swig on
+```xenial``` (which is version 3.0.8 of swig) working.  You will need
+to install the lastest version from source.
+
+1. Get swig dependency: 
+
+```sudo apt install libpcre3-dev```
+
+2. Visit [swig.org](swig.org) to get download link
+
+3. Do the installation:
+
+       tar zxvf ~/swig-3.0.12.tar.gz
+       cd swig-3.0.12
+       ./configure
+       make
+       sudo make install
+
+### Containerizing
 
-## Containerizing (WARNING: these instructions are likely obsolete with python3 port) 
+(WARNING: instructions are obsolete with the Python 3 port) 
 
 To make pypop more portable (given that some of its dependencies are currently
 obsolete), it is possible to build a Singularity container which contains a