Skip to content

Update infrastructure document and move some content to contributing #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Update infrastructure document and move some content to contributing #44

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b5268cc
move some content to contributing page
adrn Jan 6, 2022
890d6c0
update infrastructure page
adrn Jan 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions src/contributing/contributing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,38 @@ git push origin Spectral-Line-Fitting

When the tutorial is ready for broader community feedback, [open a pull request](https://help.github.com/articles/creating-a-pull-request/) against the main `astropy-tutorials` repository in order for the community to review the new tutorial.

## Marking a cell with an intentional error

Edit the cell metadata of the cell in which you would like to raise an exception and add the following to the top-level JSON: `"tags": ["raises-exception"]` This tag is recognized by the latest (master) version of nbconvert.

## Automatically Strip Output and Notebook Metadata

Jupyter notebooks contain some metadata that is typically hidden from users, which contains, for example, information about the Python kernel used to run it, the version of IPython, etc. When tutorial authors or maintainers edit notebooks, this metadata is automatically modified by Jupyter, leading to superfluous and sometimes confusing changes to the notebooks when viewed in a "diff" locally or on GitHub.

In order to prevent such metadata updates from appearing in pull requests, we therefore recommend that any contributor or maintainer install and use [nbstripout](https://github.com/kynan/nbstripout) set up with an automatic Git hook to clean metadata changes whenever a notebook change is committed to your local repo. To install `nbstripout`, first pip install it with:

```
pip install nbstripout
```

This repo is already configured. Next, configure Git within your local clone of astropy-tutorials to tell `nbstripout` to intervene whenever you commit changes in the repo. To do this, you first have to "install" it with:

```
nbstripout --install
```

Then, to tell `nbstripout` to ignore metadata changes, you must also run:

```
git config filter.nbstripout.extrakeys '
metadata.celltoolbar metadata.kernel_spec.display_name
metadata.kernel_spec.name metadata.language_info.codemirror_mode.version
metadata.language_info.pygments_lexer metadata.language_info.version
metadata.toc metadata.notify_time metadata.varInspector
cell.metadata.heading_collapsed cell.metadata.hidden
cell.metadata.code_folding cell.metadata.tags cell.metadata.init_cell'
```

## Data Files

### For tutorial authors
Expand Down
109 changes: 20 additions & 89 deletions src/contributing/infrastructure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,111 +1,42 @@
---
title: 'About the Tutorials Infrastructure'
title: 'About the Learn Astropy Infrastructure'
slug: '/contributing/infrastructure'
---

## Overview
# Overview

Tutorials are written as Jupyter notebooks on the `master` branch of the `astropy/astropy-tutorials` repository in `tutorials/notebooks/`. These notebook files do not contain output in order to simplify version-controlling the files.
The main interface to accessing tutorials and guides developed by the Astropy community is through the [Learn Astropy website](https://learn.astropy.org). This website provides a dynamic search bar that enables searching educational resources that are indexed by Learn Astropy, but it is not the main location where this material is hosted. For example, the Astropy tutorials are written as Jupyter notebooks in the [astropy-tutorials](https://github.com/astropy/astropy-tutorials) repository, and the guides are written as individual Jupyter books in the Astropy organization (e.g., [the CCD reduction guide](https://github.com/astropy/ccd-reduction-and-photometry-guide)). More information about the infrastructure that renders the tutorials and guides is given below.

The rendered Astropy-tutorials site is built using Sphinx with the Astropy theme to look like the main documentation. Sphinx requires restructured text (RST) files for its build process, so use an intermediate step to run the notebooks to produce output, and then convert the notebook files into RST files.
## Astropy Tutorials

We use [CircleCI](https://circleci.com) to do the Sphinx build, which is then pushed to the `gh-pages` branch of the repository and served by GitHub. The notebooks are first converted to RST files during the Sphinx build by doing the conversion at the end of the [Sphinx configuration file](https://github.com/astropy/astropy-tutorials/blob/master/tutorials/conf.py).
The Astropy tutorials are written as Jupyter notebooks and stored in subdirectories within the `tutorials` folder of the [astropy-tutorials](https://github.com/astropy/astropy-tutorials) repository. These notebook files do not contain output in order to reduce the file size burden of version-controlling the files, but the notebooks are executed and rendered as static HTML pages and then ingested into the Learn Astropy website.

## Why not use nbsphinx?
The rendered Astropy tutorials are generated using [nbcollection](https://github.com/astropy/nbcollection), which is a lightweight wrapper around [nbconvert](https://nbconvert.readthedocs.io/) that handles executing and converting nested directory structures of Jupyter notebook files.

Both running and converting the notebooks is handled automatically by the Sphinx plugin `nbsphinx`, but it doesn't support all of the features we want. In particular, there is no supported way to modify the template file that controls the output RST file that gets generated from each notebook; we want to be able to modify the template so we can add the links mentioned above.
We use GitHub actions for continuous integration, which runs [nbcollection](https://github.com/astropy/nbcollection) on the tutorials in the [astropy-tutorials](https://github.com/astropy/astropy-tutorials) repository and passes on the rendered HTML pages to the Learn Astropy website build infrastructure, described below.

## Tutorials directory structure
### Tutorials Directory Structure

The notebook files must be written as a single Jupyter notebook in a directory within the `tutorials/notebooks` directory. The name of the notebook must be the same as the subdirectory name. This is just needed for auto-generating links to the source notebooks from the generated RST pages.
The notebook files are be written as individual Jupyter notebooks in subdirectories within the `tutorials` folder. One folder can contain multiple tutorial files, for example [the Astropy coordinates tutorials](https://github.com/astropy/astropy-tutorials/tree/main/tutorials/astropy-coordinates).

## Testing notebook execution
### Testing notebook build and render locally

You can use the custom nbconvert script in the astropy-tutorials repository to test that the tutorials all execute correctly. From the top-level repository path:
You can test the Astropy tutorials build machinery locally using [nbcollection](https://github.com/astropy/nbcollection) through the provided [Makefile](https://github.com/astropy/astropy-tutorials/blob/main/Makefile). For example, by running:

```
python scripts/convert.py tutorials/notebooks -v --exec-only
```
make

Running the convert script with the flag `--exec-only` will just execute the notebooks and won't generate RST files. If you have already run the notebooks once, you may need to also specify the `-o` or `--overwrite` flag: by default, the script will only execute notebooks that haven't already been executed. The `-v` flag just tells the script to output more "verbose" messages, which you may or may not want.
in the root of the cloned `astropy-tutorials` repository, this will (1) check your local Python environment to make sure you have installed all dependencies (specified in the [pip requirements file](https://github.com/astropy/astropy-tutorials/blob/main/pip-requirements.txt)), (2) execute the notebooks using `nbcollection`/`nbconvert`, (3) convert the executed notebooks to HTML using `nbcollection`/`nbconvert`. You can run these commands separately, for example, to just execute the tutorials:

The above command will execute all notebooks in any subdirectory of the `tutorials/notebooks` path. If you want to just execute a single notebook, you can specify the path to a single notebook file, e.g.:
make execute

```
python scripts/convert.py tutorials/notebooks/coordinates/coordinates.ipynb -v --exec-only
```
Or to then convert the tutorials to HTML:

You can also do this when running and generating RST files, which can be useful when writing a new tutorial to make sure it renders in RST properly. To do this, just remove the `--exec-only` flag:
make convert

```
python scripts/convert.py tutorials/notebooks/coordinates/coordinates.ipynb -v
```
These commands will create a `build` path in the root of the repository folder that, if the commands execute successfully, will contain the executed tutorial notebooks and the converted HTML versions of each tutorial.

## Building the tutorials page locally
TODO: run only the modified ones locally??

For this, you can use the [Makefile](https://github.com/astropy/astropy-tutorials/blob/master/Makefile) at the top-level of the tutorials repository. From the root level of the cloned or downloaded repository:
## Learn Astropy Website Infrastructure

```
make html
```

Will execute and convert the Jupyter notebooks to RST files, then do the Sphinx build. If this returns an error, you may need to initialize the `astropy_helpers` submodule (read the error message). That is, you may need to do:

```
git submodule init
git submodule update
make html
```

Once this is done, you will find the index for the pages in your local `build/html/index.html` file.

For testing, you may want to run the build process on just one notebook file, as the full build takes some time to execute and convert all of the tutorial notebooks. To do this, you can set the `NBFILE` environment variable to specify the path to a notebook file relative to the `tutorials` path. For example, to run the build process on just the FITS-header tutorial:

```
$ NBFILE=notebooks/FITS-header/FITS-header.ipynb make html
```

If you use multiple environments to manage your python installation, you might be surprised to find that by default this build does _not_ use the same python environment you are running sphinx in. This is because the `nbconvert` machinery depends on Jupyter kernels to create a separate environment to run each notebook. To use a specific environment, you will need to use the `jupyter kernelspec` or `ipykernel install` command to create a named kernel for your favored environment. Then pass it into sphinx using the `NBCONVERT_KERNEL` environment variable. Something like:

```
$ python -m ipykernel install --user --name astropy-tutorials --display-name "Python (astropy-tutorials)"
$ NBCONVERT_KERNEL=astropy-tutorials make html
```

## Releases

We will release a new version of the tutorials with each major release of the Astropy core package; i.e. we will release for 3.0, 3.1, etc., but not for bugfix releases like 2.0.3, etc. With each release, we update the pinned versions of the global dependency files (`conda-envirionment.yml` for Anaconda and `pip-requirements.txt` for pip).

To actually update the version, modify the `metadata.cfg` at the root of this repository with the new version. This is the version number that will be shown in the sphinx builds. Note that if it ends in `.dev`, this will be followed by a revision number that is determined by the number of git commits.

## Marking a cell with an intentional error

Edit the cell metadata of the cell in which you would like to raise an exception and add the following to the top-level JSON: `"tags": ["raises-exception"]` This tag is recognized by the latest (master) version of nbconvert.

## Automatically Strip Output and Notebook Metadata

Jupyter notebooks contain some metadata that is typically hidden from users, which contains, for example, information about the Python kernel used to run it, the version of IPython, etc. When tutorial authors or maintainers edit notebooks, this metadata is automatically modified by Jupyter, leading to superfluous and sometimes confusing changes to the notebooks when viewed in a "diff" locally or on GitHub.

In order to prevent such metadata updates from appearing in pull requests, we therefore recommend that any contributor or maintainer install and use [nbstripout](https://github.com/kynan/nbstripout) set up with an automatic Git hook to clean metadata changes whenever a notebook change is committed to your local repo. To install `nbstripout`, first pip install it with:

```
pip install nbstripout
```

This repo is already configured. Next, configure Git within your local clone of astropy-tutorials to tell `nbstripout` to intervene whenever you commit changes in the repo. To do this, you first have to "install" it with:

```
nbstripout --install
```

Then, to tell `nbstripout` to ignore metadata changes, you must also run:

```
git config filter.nbstripout.extrakeys '
metadata.celltoolbar metadata.kernel_spec.display_name
metadata.kernel_spec.name metadata.language_info.codemirror_mode.version
metadata.language_info.pygments_lexer metadata.language_info.version
metadata.toc metadata.notify_time metadata.varInspector
cell.metadata.heading_collapsed cell.metadata.hidden
cell.metadata.code_folding cell.metadata.tags cell.metadata.init_cell'
```
TODO: briefly describe algolia and website infrastructure.