Convert all notebooks to sphinx gallery format

[StFroese]

As discussed in #2221

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[StFroese]

@maxnoe do you know why the docs build is failing? I only see Warnings in the log of the CI but no errors

[StFroese]

nvm, fixed it. Now it's failing because of magic commands

[StFroese]

fmt: skip doesn't work, any ideas?

(cta-dev) ~/D/p/c/examples ❯❯❯ black --preview examples/core/containers.py                                       ✘ 146 sphinx-gallery-notebooks ✭ ✚ ◼
error: cannot format examples/core/containers.py: Cannot parse: 90:0: ?EventContainer # fmt: skip

Oh no! 💥 💔 💥
1 file failed to reformat.

[StFroese]

@maxnoe @kosack Does one of you know how to add sphinx-gallery to the readthedocs CI?

[maxnoe]

You need to add the needed packages to the docs extra in setup.cfg

[kosack]

I do have a problem building the environment for python 3.11:

$ mamba env create -n cta-v0.19-newdocs -f environment.yml
Could not solve for environment specs
Encountered problems while solving:
  - package numba-0.56.2-py310h62db5c2_0 requires python_abi 3.10.* *_cp310, but none of the providers can be installed

With the main branch, I can install 3.11, so it must have something to do with sphinx's dependencies. With 3.10 it works

[StFroese]

With the main branch, I can install 3.11, so it must have something to do with sphinx's dependencies

I get the same error when I switch to the main branch and set python=3.11 in the environment.yml

[kosack]

I get the same error when I switch to the main branch and set python=3.11 in the environment.yml

Could be a macos issue: I installed the ctapipe env with mamba and 3.11 a while ago, and haven't had issues. However, it could have changed if something updated in the mean time. I opened #2379 to check.

With 3.10, I also can't build the docs on macos, but not sure if that is a general issue. The file "theta_square.py" fails with hundreds of errors like:

/Users/kkosack/Projects/CTA/Working/ctapipe/ctapipe/tools/train_particle_classifier.py:docstring of ctapipe.tools.train_particle_classifier.TrainParticleClassifier.examples:: WARNING: py:class reference target not found: t.Union
/Users/kkosack/Projects/CTA/Working/ctapipe/ctapipe/tools/train_particle_classifier.py:docstring of ctapipe.tools.train_particle_classifier.TrainParticleClassifier.examples:: WARNING: py:class reference target not found: Unicode
/Users/kkosack/Projects/CTA/Working/ctapipe/ctapipe/tools/train_particle_classifier.py:docstring of ctapipe.tools.train_particle_classifier.TrainParticleClassifier.name:: WARNING: py:class reference target not found: t.Union
/Users/kkosack/Projects/CTA/Working/ctapipe/ctapipe/tools/train_particle_classifier.py:docstring of ctapipe.t

[kosack]

The galleries seem to be missing all outputs and images.

And cells which should have output in between are joined and no output is generated:

[StFroese]

Ah yes, I was busy with compiling this without errors. I will take a look.

[StFroese]

Some pretty funny stuff is happening that I don't understand :D

So this is currently failing since Tools.ipynb gets stuck in an infinite loop.
I found out the following things (clone this branch and build the docs to reproduce otherwise you won't have the same directories):

1. If I copy Tools.ipynb and Tools.json from docs/examples/core to examples/examples/core the tool gets stuck in this cell

try:
    tool2.run(argv=["--config", "Tools.json"])
except SystemExit as e:
    assert e.code == 0, f"Tool returned with error status {e}"

The debug output prints some stuff about 2023-07-11 09:39:45,239 INFO [builtins.mytool] which already seems odd since this is not a builtin.
It also states bad value: The 'iterations' trait of a MyTool instance expected an int, not the str 'badval'.. Setting the iterations to badval belongs to a cell above and this happens in a try-except block, this should actually never happen.

2. If I copy Tools.ipynb and Tools.json from docs/examples/core to examples/examples or create a new folder examples/examples/examples, the code runs without any problem and the debug output states something like 2023-07-11 09:41:51,286 INFO [__main__.mytool]

@maxnoe, @kosack do you have any idea what's going on?

[maxnoe]

Is there another python script in the core directory that might shadow another module? Especially a standard library module?

[StFroese]

examples/examples/core:

-rw-r--r--   1 stefan  staff      5246 Jul  7 16:09 InstrumentDescription.py
-rw-r--r--   1 stefan  staff        66 Jul  7 16:09 README.rst
-rw-r--r--   1 stefan  staff    183546 Jul 11 09:59 Tools-thisbranch.ipynb
-rw-r--r--   1 stefan  staff       158 Jul  7 16:11 Tools.json
-rw-r--r--   1 stefan  staff     10125 Jul  7 16:11 Tools.py
-rw-r--r--   1 stefan  staff     13533 Jul  7 16:10 container.h5
-rw-r--r--   1 stefan  staff      5852 Jul  7 16:09 containers.py
-rw-r--r--   1 stefan  staff  32022561 Jul 11 09:39 mytool.provenance.log
-rw-r--r--   1 stefan  staff      3212 Jul  7 16:09 provenance.py
-rw-r--r--   1 stefan  staff      7941 Jul  7 16:09 table_writer_reader.py

[StFroese]

the problem seems to be having a file Tools.py in the same directory as Tools.ipynb.

[StFroese]

@maxnoe, @kosack pls review again :)

The %%timeit cells don't work in the documentation but they are executable in the generated .ipynb notebooks.

[maxnoe]

Some pretty funny stuff is happening that I don't understand :D

We figured out what happened. The converted notebook file is called Tools.py and the config file we give to the tool is called Tools.json.

Traitlets loads all files in the directory with the same basename and extensions it supports for configuration files. So in the notebook, the tool tried to load the notebook itself as config file, resulting in an endless loop.

I opened an issue here: ipython/traitlets#850

For now I'd just propose to call the config file config.json or config.yaml instead of Tools.yaml.

[kosack]

The animations don't seem to render correctly. See here for example

In the old docs, this appears as an HTML5 video

[StFroese]

@kosack animations are rendering now

[StFroese]

@maxnoe done

[maxnoe]

There is a weird double examples/examples structure now and things that are rendered as part of the docs are living outside of the docs/ directory, I don't like that very much.

Why move out these things from where they are also in the toc hierarchy? Makes finding where things are defined much harder.

Also, I'd prefer to have the title be Tutorials and Examples and have the Tutorials come first.

[kosack]

Minor style issue: this headline is at the section level, so the ones below it (each example/tutorial) should be a level deeper (so subsection)

For RST:

# with overline, for parts
* with overline, for chapters
= for sections
- for subsections
^ for subsubsections
" for paragraphs

[kosack]

That would fix the wrong hierarchy here:

[maxnoe]

Why is this a README.txt file and not a .rst file called index?

[StFroese]

It's just the default sphinx gallery style

By default, Sphinx-Gallery creates all the files that are written in the sphinx-build directory, either by generating rst and images from a *.py in the gallery-source, or from creating index.rst from README.txt in the gallery-source.

I think I can change it

You may also want to pass raw rst from the gallery-source to the sphinx-build[...] To accommodate this, you may set copyfile_regex in sphinx_gallery_conf. The following copies across rst files.

sphinx_gallery_conf = {
    ...
   'copyfile_regex': r'.*\.rst',
}

[StFroese]

Maybe the copyfile_regex is not needed since source and build dir are the same in this case

[StFroese]

But if I do this the toctree will not be generated by sphinx-gallery

[maxnoe]

By default, Sphinx-Gallery creates all the files that are written in the sphinx-build directory,

Why did you change that so that the generated files clutter the docs source directory?

[maxnoe]

Thanks a lot!

[maxnoe]

Ok, maybe we do need to move the scripts out of the docs directory. It seems the "doctest-modules" hangs on plots being opened because it tries to run the examples.

[maxnoe]

@kosack @StFroese ok, I think this is ready now