Updated checks and CI

Author: CallumWalley

.github/workflows/README.md

@@ -2,11 +2,35 @@
 
 Description of current CI workflow.
 
+## [fetch_includes.yml](fetch_includes.yml)
+
+Retrieves dynamically generated content from external sources.
+
+Currently retrieves:
+
+- Software module list from [modules-list](https://github.com/nesi/modules-list).
+- Glossary, spellcheck dictionary and snippets from [nesi-wordlist](https://github.com/nesi/nesi-wordlist)
+
+It then runs [link_apps_pages.py](#link_apps_pagespy).
+
+All modified files are added to a new branch called `new-assets` and merged into main.
+
+In theory, all this could be done at deployment, but I wanted to make sure that changes to these remote files didn't break anything.
+
+## [link_apps_pages.py](link_apps_pages.py)
+
+A Python script used to add a link to the appropriate documentation to [modules-list.json](../../docs/assets/module-list.json).
+
+The script checks all titles of input files, and sets the `support` key to be equal to the pages url.
+It also adds whatever tags are on that page to the `domains` key.
+
+_One day I would like to simplify this whole thing._
+
 ## [checks.yml](checks.yml)
 
 A series of QA checks run on the documentation.
 
-The checks can be started manually from the ![workflow page](https://github.com/nesi/agdr-docs/actions/workflows/checks.yml/badge.svg),
+The checks can be started manually from the ![workflow page](https://github.com/nesi/support-docs/actions/workflows/checks.yml/badge.svg),
 select the target branch, give the pattern of files to include, and select which checks you want done.
 
 Checks will also be run on any _non main_ branch pushes. All checks will be run, but only on _changed_ files.
@@ -16,7 +40,3 @@ More info on what these checks do in [README.md](../../checks/README.md)
 ## [deploy.yml](deploy.yml)
 
 Runs on push to _main_ branch. Builds and deploys pages.
-
-## [depo_deploy.yml](demo_deploy.yml)
-
-## [auto_merge.yml](auto_marge.yml)

.github/workflows/auto_merge.yml

@@ -9,14 +9,14 @@ env:
   LABEL_NAME: auto_merge
 jobs:
   automerge:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
     - uses: actions/checkout@v4
       with:
         fetch-depth: 0
     - run: |
         for branch in $(gh pr list -R ${TARGET_REPO} --label "${LABEL_NAME}" --json headRefName --jq .[].headRefName);do
-            git checkout $branch
+            git checkout ${branch}
             gh pr merge --squash --auto --delete-branch
         done
 

.github/workflows/checks.yml

@@ -4,6 +4,7 @@ env:
 on:
   push:
     branches-ignore: [main]
+  repository_dispatch:
   workflow_dispatch:
     inputs:
       fileList:
@@ -27,14 +28,18 @@ on:
         description: Check Meta
         default: true
         type: boolean
+      checkSlurm:
+        description: Check Slurm
+        default: true
+        type: boolean
       testBuild:
         description: Test Build
         default: true
         type: boolean
 jobs:
   get:
     name: Determine what files to check
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - name: Checkout Repo
         uses: actions/checkout@v4
@@ -58,7 +63,7 @@ jobs:
   spellcheck:
     name: Check Spelling
     if: ${{ github.event_name != 'workflow_dispatch' || inputs.checkSpelling }}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: get
     steps:
       - if: ${{! needs.get.outputs.filelist}}
@@ -80,7 +85,7 @@ jobs:
   proselint:
     name: Check Prose
     if: ${{github.event_name != 'workflow_dispatch'|| inputs.checkProse}}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: get
     steps:
       - if: ${{! needs.get.outputs.filelist}}
@@ -100,7 +105,7 @@ jobs:
   mdlint:
     name: Check Markdown
     if: ${{github.event_name != 'workflow_dispatch' || inputs.checkMarkdown}}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: get
     steps:
       - if: ${{! needs.get.outputs.filelist}}
@@ -120,7 +125,7 @@ jobs:
   metacheck:
     name: Check page meta
     if: ${{github.event_name != 'workflow_dispatch'|| inputs.checkMeta}}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: get
     steps:
       - if: ${{ ! needs.get.outputs.filelist}}
@@ -134,10 +139,27 @@ jobs:
         run: |
           shopt -s globstar extglob
           python3 checks/run_meta_check.py ${{needs.get.outputs.filelist}}
+  slurmcheck:
+    name: Check slurm scripts
+    if: ${{github.event_name != 'workflow_dispatch'|| inputs.checkSlurm}}
+    runs-on: ubuntu-22.04
+    needs: get
+    steps:
+      - if: ${{ ! needs.get.outputs.filelist}}
+        name: No files to check meta on.
+        run: exit 0
+      - if: ${{needs.get.outputs.filelist}}
+        name: Check out repo.
+        uses: actions/checkout@v4
+      - if: ${{needs.get.outputs.filelist}}
+        name: Check markdown meta.
+        run: |
+          shopt -s globstar extglob
+          python3 checks/run_slurm_lint.py ${{needs.get.outputs.filelist}}
   testBuild:
     name: Test build
     if: ${{github.event_name != 'workflow_dispatch' || inputs.testBuild}}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     needs: get
     steps:
       - uses: actions/checkout@v4

.github/workflows/demo_deploy.yml

@@ -14,7 +14,7 @@ jobs:
   demo-deploy:
     continue-on-error: true
     name: Trigger test deployments
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - name: Trigger Workflow in Another Repository
         run: |
@@ -37,14 +37,6 @@ jobs:
       #     github_token: ${{ secrets.PAT }}
       #     workflow_file_name: deploy.yml
       #     client_payload: '{"targets":"${GITHUB_REPOSITORY}:${HEAD}", "use-cache":"true"}'
-      - name: Wait for Workflow Action
-        run: |
-          curl -L \
-              -X POST \
-              -H "Accept: application/vnd.github+json" \
-              -H "Authorization: Bearer ${{ secrets.PAT }}" \
-              -H "X-GitHub-Api-Version: 2022-11-28" \
-              https://api.github.com/repos/${TARGET_OWNER}/${TARGET_REPO}/actions/runs/deploy.yml
       - name: Wait for Workflow Action
         run: |
           # Just give a minute or so to deploy

.github/workflows/deploy.yml

@@ -1,8 +1,8 @@
 name: Deploy to gh-pages 
 on: 
- push:
+  push:
     branches: [main]
- workflow_dispatch:
+  workflow_dispatch:
 env:
   PYTHON_VERSION: 3.x
   GH_TOKEN: ${{ github.token }}
@@ -14,7 +14,7 @@ permissions:
 jobs:
   documentation:
     name: Build documentation
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - name: Checkout repository
         uses: actions/checkout@v4
@@ -23,6 +23,7 @@ jobs:
           sparse-checkout: |
             docs
             overrides
+            .github
       - name: Set up Python runtime
         uses: actions/setup-python@v5
         with:

checks/README.md

@@ -16,12 +16,6 @@ Spellcheck pipeline settings can be modified in [.spellcheck.yml](../.spellcheck
 List of custom words can be found in [dictionary.txt](../docs/assets/glossary/dictionary.txt),
 however you **should not edit this manually**, see [adding-words-to-dictionary](../docs/CONTRIBUTING.md#adding-words-to-dictionary).
 
-This list is automatically updated daily in the case of docs.nesi, but not the other sites.
-You should occasionally run 
-`wget -O docs/assets/glossary/dictionary.txt https://raw.githubusercontent.com/nesi/nesi-wordlist/main/outputs/dictionary.txt`
-to keep it up to date.
- 
-
 ### Limitations
 
 Spellchecker does not provide output lineumber / column.

checks/fail_checks.md

@@ -0,0 +1,70 @@
+---
+created_at: 2025-01-28
+template: not_a_template
+---
+
+
+
+This page is meant for testing the linting checks. For all checks to run properly this file should be moved under `docs`.
+
+It should trigger the fail threshold for `meta-checks`, `proselint-checks`, `spelling-checks`, `slurm-lint`, and `test-build`
+
+Add tests here as needed.
+
+Somee tpyos for spelchecker to triger fail neeeds to be at laest fiveteen keybored whopsers whiche is qiute a hihe tolerence i thinc. I woold loweir this threshehold bute for the momenent it can be a bitte overzaelos.
+
+``` as
+typos shuold be igonred in code blokcs 
+```
+
+!!! warning "some admonistion"
+    contents of admonition
+
+Typos should `be igonred` in inline code blocks.
+
+Typos should [be ignored](https://www.docs.nesi.org.nz)
+Typos should [be ignored](../docs/General/FAQs/How_do_I_request_memory.md)
+
+
+
+Bad formatting for markdownlint 
+
+  * bullet points
+- bullet points
+
+### Improper
+
+## Header nesting
+ 
+
+```sl
+!#/bin/bash
+
+
+#SBATCH -j short-option
+#SBATCH --not-a-real-flag=not_real
+
+module load something
+
+```
+
+[bad link](../docs/General/not-a-page.md)
+
+[bad internal link](#impropers)
+
+Very very bad prose shit this is probably very unproper language sorry about that hopefully the chairman doesn't see lets circle back around.
+
+The following checks will only work if moved into a subdir of `docs/`
+
+https://www.nesi.org.nz/deadlink
+
+
+{{broken_macro()}}
+
+{% include "not-real.md" -%}
+
+
+{{broken_macro()}}
+
+
+{% set app_name = page.no_a_variable | trim %}