Run notebooks on RTD with sphinx-gallery and add WorkGraph tutorials #474

agoscinski · 2024-09-11T19:24:54Z

Similar as in aiida-workgraph aiidateam/aiida-workgraph#234 I would like to integrate an execution of the notebooks as the notebooks become quickly outdated otherwise. This simplifies the update process of the notebooks as they will be automatically run with each CI, so errors can be immediately seen and then fixed. Since we already adapted the tutorials from the aiida-workgraph to the sphinx-scripts and I would like to move these tutorials out of aiida-workgraph to this repo as fast as possible I focused on moving these as a first example. Nevertheless I would like to adapt all tutorials to be executable in the future.

I tried myst_nb for the execution as it was already provided by the repo, but I ran into issues that ran explained below, so I used the same strategy as in aiida-workgraph by using sphinx-gallery.

Note that the sections Writing workflows and Writing workflows with WorkGraph (the new section including the WorkGraph) have overlaps that could be better resolved, but I did not want to spend the time in the first PR setting up the project structure, since I feel like this could become very time consuming.

Also Note that the CI link checker has been merged to RTD. Since for link checking one also needs to run the tutorials, I thought this is less maintenance as one just needs to update the RTD setup. This also does not affect the building time, because on the second build when creating the html files, the files from the first build can be Reused. Compare the times of https://readthedocs.org/projects/aiida-tutorials/builds/25605020/ of two builds (linkcheck plus html) and https://readthedocs.org/projects/aiida-tutorials/builds/25603891/ one build (only html)

…ying.ipynb since it depends on the execution of other notebooks

agoscinski · 2024-09-12T03:26:07Z

I tried using myst_nb but I get WARNING: No source code lexer found for notebook cell 1 [myst-nb.lexer] warnings which I cannot find any solution to online. It seems that I need to change markdown part in the cells, but I am really not sure. Also when an error is thrown it is written into a file and not printed out, which is really crucial for debugging runs on RTD. So I am switching to sphinx-gallery.

…ogen files; add conda action to install environment.yml

agoscinski · 2024-09-12T12:31:00Z

.github/workflows/ci.yml

+    # only pushes to main trigger
+    branches: [main]
+  pull_request:
+    # always triggered


This change is unrelated to this PR. It stops that the CI is run two time for each PR. I will put this in a separate PR

agoscinski · 2024-09-12T12:32:30Z

.github/workflows/ci.yml

@@ -11,29 +16,8 @@ jobs:

    steps:
    - uses: actions/checkout@v2
-    - name: Set up Python 3.12
+    - name: Set up Python 3.11


To build aiida-core in CI, I needed to use Python 3.11. For pre-commit it is not needed but I think we should just stick with Python 3.11 if there are issues with Python 3.12.

agoscinski · 2024-09-12T12:32:50Z

.github/workflows/ci.yml

    - uses: pre-commit/[email protected]
-
-  build:


This is now run in the RTD, check out the .readthedocs.yml

.readthedocs.yml

README.md

agoscinski · 2024-09-12T12:34:50Z

docs/conf.py

@@ -68,7 +69,7 @@
 ipython_mplbackend = ""

 copybutton_selector = "div:not(.no-copy)>div.highlight pre"
-copybutton_prompt_text = ">>> |\.\.\. |\$ |In \[\d*\]: | {2,5}\.\.\.: | {5,8}: "
+copybutton_prompt_text = ">>> |... |$ |In [d*]: | {2,5}...: | {5,8}: "


The escapes raised some warnings

agoscinski · 2024-09-12T12:35:05Z

docs/conf.py

+    "https://www.big-map.eu/",
+    r".*concept/index.html",
+    r".*howto/index.html",
+    "http://127.0.0.1:8000/workgraph",


Here we ignore workgraph related files

docs/conf.py

agoscinski · 2024-09-12T12:36:04Z

docs/conf.py

+from pathlib import Path
+
+
+def copy_html_files(app, exception):


copy paste from aiida-workgraph to move html files from the sphinx gallery folder to the sphinx source, so we can display the widgets

agoscinski · 2024-09-12T12:36:47Z

docs/gallery/sections/writing_workflows_with_workgraph/autogen/GALLERY_HEADER.rst

+
+================================
+Running workflows with WorkGraph
+================================


sphinx-gallery requires a GALLERY_HEADER.rst but we do not use if anywhere therefore the :orphan: at the top to surppres warnings

agoscinski · 2024-09-12T12:37:06Z

docs/gallery/sections/writing_workflows_with_workgraph/autogen/eos.py

@@ -0,0 +1,283 @@
+"""


copy paste from aiida-workgraph tutorials as all sphinx-gallery scripts

requirements.txt

agoscinski · 2024-09-12T12:38:20Z

requirements.txt

+# to run notebooks
+aiida-quantumespresso
+aiida-pseudo
+aiida-workgraph[widget]


I think aiida-workgraph we do not want to pin, but I am not sure about the others

agoscinski · 2024-09-12T12:41:54Z

docs/sections/writing_workflows_with_workgraph/index.rst

+    ^^^^^^^^^^^^
+
+    A short module on how to write the basic type of workflows in AiiDA: work functions.
+    The module also revises the usage of calculation functions to add simple Python functions to the provenance.


This text needs some rework, maybe one of the reviewer has some ideas

agoscinski · 2024-09-12T12:41:59Z

docs/sections/writing_workflows_with_workgraph/index.rst

+    ^^^^^^^^^^^^
+
+    A step-by-step introduction to the basics of writing work chains in AiiDA.
+    After completing this module, you will be ready to start writing your own scientific workflows!


This text needs some rework, maybe one of the reviewer has some ideas

superstar54 · 2024-09-18T16:42:40Z

Hi @agoscinski , thanks for the work.

I tried myst_nb for the execution as it was already provided by the repo, but I ran into issues that ran explained below, so I used the same strategy as in aiida-workgraph by using sphinx-gallery.

What's the issue? Can we fix it? It's better to use the same method for all, to make it easy to maintain.

Refactor the notebooks

There are duplicate contents in the three notebooks:

EOS workgraph in both zero-to-hero and the eos notebooks.
atomization energy workgraph in both zero-to-hero and computational materials science notebooks.
I would suggest replacing the zero-to-hero with workgraph basic.

Compare with WorkChain

Since the previous section is WorkChain, and it also shows examples of simple add and real-world EOS:

There are some calcfunctions already defined in the WorkChain section; use them, if possible, to ensure consistency.
Mention the WorkChain version, or try to compare them with WorkGraph, so that users know why WorkGraph makes it simple.

Use same style as the other section

The Workchain section is well-written, explaining steps step by step and providing extensive descriptions. It also has Note, Important and explanation. I suggest using the same style to make the website more uniform.
Also, the font size of the title/subtitle, etc., should be the same.

To summarize, I would suggest rewriting the WorkGraph section to fit into the aiida-tutorial better.

superstar54 · 2024-09-18T16:45:16Z

docs/gallery/sections/writing_workflows_with_workgraph/autogen/zero_to_hero.py

+# What's Next
+# ===========
+#
+# +-----------------------------------------------------+-----------------------------------------------------------------------------+
+# | `Concepts <../../concept/index.html>`_              | A brief introduction of WorkGraph’s main concepts.                          |
+# +-----------------------------------------------------+-----------------------------------------------------------------------------+
+# | `HowTo <../../howto/index.html>`_                   | Advanced topics, e.g., flow control using `if`, `while`, and `context`.     |
+# +-----------------------------------------------------+-----------------------------------------------------------------------------+
+#


This is not valid anymore; maybe remote it.
Similar to the WorkChain, we may add some exercises.

agoscinski · 2024-09-19T06:09:40Z

What's the issue? Can we fix it? It's better to use the same method for all, to make it easy to maintain.

Hey Xing I wrote it in the second post.

I tried using myst_nb but I get WARNING: No source code lexer found for notebook cell 1 [myst-nb.lexer] warnings which I cannot find any solution to online. It seems that I need to change markdown part in the cells, but I am really not sure. Also when an error is thrown it is written into a file and not printed out, which is really crucial for debugging runs on RTD. So I am switching to sphinx-gallery.

Note that myst_nb is also only used for one notebook here and not well integrated.

Thanks for the suggestions Xing, but this PR is not intended to refactor the existing examples as I mentioned.

Note that the sections Writing workflows and Writing workflows with WorkGraph (the new section including the WorkGraph) have overlaps that could be better resolved, but I did not want to spend the time in the first PR setting up the project structure, since I feel like this could become very time consuming.

We can put your comments in an issue.

agoscinski added 7 commits September 11, 2024 21:23

WIP: run notebooks on RTD

1298f45

add deps

cd0c49b

add installation of req

8631643

introduce SPHINX_LINKCHECK to not execute for linkchecks, ignore quer…

f25fe8e

…ying.ipynb since it depends on the execution of other notebooks

try to specify requirements in conda environment.yml

7430793

integrate workgraph tutorials

4c204df

debug linkcheck

db2e36a

agoscinski added 22 commits September 12, 2024 09:23

initialize sphinx-gallery

ddbc440

debug sphinx linkcheck

1ea66fe

fix warnings

4bdba12

need to build the sphinx doc because the index.rst references the aut…

2b2c43e

…ogen files; add conda action to install environment.yml

add nicer front page

e52d86c

add imports to conf.py

5b48209

update conda version

8e67630

pre-commit run

4db0691

use latest

1fa3aaf

update CI run policy, it was run twice before

798304e

maybe auto update solves libmamba solver issue

c87d18f

use minifore + mamba to see if libmambas solver is there

adfd2c0

fix path

86731fd

add copy html files to setup

102d5e8

pre-commit run

42ff514

downgrade to python 3.11 due to some conflicts

43f6116

update README

12e2872

debug ci

3a2c005

pre-commit run

8049b8c

debug ci

ef349ac

debug ci

0a06c78

debug ci

15fb57a

agoscinski added 7 commits September 12, 2024 12:08

debug ci

8eb5d0f

debug ci

6118d30

debug ci

e38a372

move linkcheck to RTD

32bde7f

debug rtd

7620580

add html to link ignore, also add_css_file

e531860

add regex correctly

55210f2

agoscinski changed the title ~~WIP: run notebooks on RTD~~ Run notebooks on RTD with sphinx-gallery and add WorkGraph tutorials Sep 12, 2024

agoscinski requested review from GeigerJ2 and superstar54 September 12, 2024 12:30