To avoid pointless conflicts while working with jupyter notebooks (with different execution counts or cell metadata), it is recommended to clean the notebooks before committing anything (done automatically if you install the git hooks with nbdev_install_hooks). The following functions are used to do that.
Preserve the list of keys in the main notebook metadata
+
+
+
allowed_cell_metadata_keys
+
list
+
None
+
Preserve the list of keys in cell level metadata
+
+
+
clean_ids
+
bool
+
True
+
Remove ids from plaintext reprs?
+
+
+
+
Jupyter adds a trailing to images in cell outputs. Vscode-jupyter does not.
+Notebooks should be brought to a common style to avoid unnecessary diffs:
+
+
test_nb = read_nb('../../tests/image.ipynb')
+assert test_nb.cells[0].outputs[0].data['image/png'][-1] =="\n"# Make sure it was not converted by acccident
+clean_nb(test_nb)
+assert test_nb.cells[0].outputs[0].data['image/png'][-1] !="\n"
+
+
The test notebook has metadata in both the main metadata section and contains cell level metadata in the second cell:
Passing clean_ids=True removes ids from plaintext repr outputs, to avoid notebooks whose contents change on each run since they often lead to git merge conflicts. For example:
+
<PIL.PngImagePlugin.PngImageFile image mode=L size=28x28 at 0x7FB4F8979690>
Clean all notebooks in fname to avoid merge conflicts
+
+
+
+
+
+
+
+
+
+
+
Type
+
Default
+
Details
+
+
+
+
+
fname
+
str
+
None
+
A notebook name or glob to clean
+
+
+
clear_all
+
bool
+
False
+
Remove all cell metadata and cell outputs?
+
+
+
disp
+
bool
+
False
+
Print the cleaned outputs
+
+
+
stdin
+
bool
+
False
+
Read notebook from input stream
+
+
+
+
By default (fname left to None), all the notebooks in config.nbs_path are cleaned. You can opt in to fully clean the notebook by removing every bit of metadata and the cell outputs by passing clear_all=True.
+
If you want to keep some keys in the main notebook metadata you can set allowed_metadata_keys in settings.ini. Similarly for cell level metadata use: allowed_cell_metadata_keys. For example, to preserve both k1 and k2 at both the notebook and cell level adding the following in settings.ini:
This cleans notebooks on-save to avoid unnecessary merge conflicts. The easiest way to install it for both Jupyter Notebook and Lab is by running nbdev_install_hooks. It works by implementing a pre_save_hook from Jupyter’s file save hook API.
Path to package root (default: repo with - replaced by _)
+
+
+
doc_path
+
Path
+
_docs
+
Path to rendered docs
+
+
+
tst_flags
+
str
+
notest
+
Test flags
+
+
+
version
+
str
+
0.0.1
+
Version of this release
+
+
+
doc_host
+
str
+
https://%(user)s.github.io
+
Hostname for docs
+
+
+
doc_baseurl
+
str
+
/%(repo)s
+
Base URL for docs
+
+
+
keywords
+
str
+
nbdev jupyter notebook python
+
Package keywords
+
+
+
license
+
str
+
apache2
+
License for the package
+
+
+
copyright
+
str
+
None
+
Copyright for the package, defaults to ‘current_year onwards, author’
+
+
+
status
+
str
+
3
+
Development status PyPI classifier
+
+
+
min_python
+
str
+
3.7
+
Minimum Python version PyPI classifier
+
+
+
audience
+
str
+
Developers
+
Intended audience PyPI classifier
+
+
+
language
+
str
+
English
+
Language PyPI classifier
+
+
+
recursive
+
bool_arg
+
True
+
Include subfolders in notebook globs?
+
+
+
black_formatting
+
bool_arg
+
False
+
Format libraries with black?
+
+
+
readme_nb
+
str
+
index.ipynb
+
Notebook to export as repo readme
+
+
+
title
+
str
+
%(lib_name)s
+
Quarto website title
+
+
+
allowed_metadata_keys
+
str
+
+
Preserve the list of keys in the main notebook metadata
+
+
+
allowed_cell_metadata_keys
+
str
+
+
Preserve the list of keys in cell level metadata
+
+
+
jupyter_hooks
+
bool_arg
+
False
+
Run Jupyter hooks?
+
+
+
clean_ids
+
bool_arg
+
True
+
Remove ids from plaintext reprs?
+
+
+
clear_all
+
bool_arg
+
False
+
Remove all cell metadata and cell outputs?
+
+
+
cell_number
+
bool_arg
+
True
+
Add cell number to the exported file
+
+
+
put_version_in_init
+
bool_arg
+
True
+
Add the version to the main init.py in nbdev_export
+
+
+
+
It works by copying contents from the latest tagged nbdev-template release, and rendering termplate variables using the created settings.ini file. Settings can be passed via command line args; missing settings are inferred from the current git/GitHub repo, otherwise prompted for.
nbdev_bump_version Increment version in settings.ini by one
+nbdev_changelog Create a CHANGELOG.md file from closed and labeled GitHub issues
+nbdev_clean Clean all notebooks in `fname` to avoid merge conflicts
+nbdev_conda Create a `meta.yaml` file ready to be built into a package, and optionally build and upload it
+nbdev_create_config Create a config file.
+nbdev_docs Create Quarto docs and README.md
+nbdev_export Export notebooks in `path` to Python modules
+nbdev_filter A notebook filter for Quarto
+nbdev_fix Create working notebook from conflicted notebook `nbname`
+nbdev_help Show help for all console scripts
+nbdev_install Install Quarto and the current library
+nbdev_install_hooks Install Jupyter and git hooks to automatically clean, trust, and fix merge conflicts in notebooks
+nbdev_install_quarto Install latest Quarto on macOS or Linux, prints instructions for Windows
+nbdev_merge Git merge driver for notebooks
+nbdev_migrate Convert all markdown and notebook files in `path` from v1 to v2
+nbdev_new Create an nbdev project.
+nbdev_prepare Export, test, and clean notebooks, and render README if needed
+nbdev_preview Preview docs locally
+nbdev_proc_nbs Process notebooks in `path` for docs rendering
+nbdev_pypi Create and upload Python package to PyPI
+nbdev_readme Create README.md from readme_nb (index.ipynb by default)
+nbdev_release_both Release both conda and PyPI packages
+nbdev_release_gh Calls `nbdev_changelog`, lets you edit the result, then pushes to git and calls `nbdev_release_git`
+nbdev_release_git Tag and create a release in GitHub for the current version
+nbdev_requirements Writes a `requirements.txt` file to `directory` based on settings.ini.
+nbdev_sidebar Create sidebar.yml
+nbdev_test Test in parallel notebooks matching `path`, passing along `flags`
+nbdev_trust Trust notebooks matching `fname`
+nbdev_update Propagate change in modules matching `fname` to notebooks that created them
+nbdev_update_license Allows you to update the license of your project.
Path to package root (default: repo with - replaced by _)
+
+
+
doc_path
+
Path
+
_docs
+
Path to rendered docs
+
+
+
tst_flags
+
str
+
notest
+
Test flags
+
+
+
version
+
str
+
0.0.1
+
Version of this release
+
+
+
doc_host
+
str
+
https://%(user)s.github.io
+
Hostname for docs
+
+
+
doc_baseurl
+
str
+
/%(repo)s
+
Base URL for docs
+
+
+
keywords
+
str
+
nbdev jupyter notebook python
+
Package keywords
+
+
+
license
+
str
+
apache2
+
License for the package
+
+
+
copyright
+
str
+
None
+
Copyright for the package, defaults to ‘current_year onwards, author’
+
+
+
status
+
str
+
3
+
Development status PyPI classifier
+
+
+
min_python
+
str
+
3.7
+
Minimum Python version PyPI classifier
+
+
+
audience
+
str
+
Developers
+
Intended audience PyPI classifier
+
+
+
language
+
str
+
English
+
Language PyPI classifier
+
+
+
recursive
+
bool_arg
+
True
+
Include subfolders in notebook globs?
+
+
+
black_formatting
+
bool_arg
+
False
+
Format libraries with black?
+
+
+
readme_nb
+
str
+
index.ipynb
+
Notebook to export as repo readme
+
+
+
title
+
str
+
%(lib_name)s
+
Quarto website title
+
+
+
allowed_metadata_keys
+
str
+
+
Preserve the list of keys in the main notebook metadata
+
+
+
allowed_cell_metadata_keys
+
str
+
+
Preserve the list of keys in cell level metadata
+
+
+
jupyter_hooks
+
bool_arg
+
False
+
Run Jupyter hooks?
+
+
+
clean_ids
+
bool_arg
+
True
+
Remove ids from plaintext reprs?
+
+
+
clear_all
+
bool_arg
+
False
+
Remove all cell metadata and cell outputs?
+
+
+
cell_number
+
bool_arg
+
True
+
Add cell number to the exported file
+
+
+
put_version_in_init
+
bool_arg
+
True
+
Add the version to the main init.py in nbdev_export
+
+
+
+
The table above also serves as a full reference of nbdev’s settings (excluding the path and cfg_name parameters which decide where the config file is saved). For more about PyPI classifiers, see Classifiers.
+
You can create a config file by passing all of the required settings via the command line, as well as any optional settings you’d like to override, for example:
If you don’t provide required settings from the command line, we’ll try to to infer them from git and GitHub. Finally, you’ll be asked to manually input any required settings that we couldn’t automatically fill in.
Searches up from path until cfg_name is found. User settings are loaded from ~/.config/nbdev/{cfg_name}. Unspecified optional settings return defaults.
It automatically returns defaults for keys not specified in the config file. Here we create an empty config file and access lib_path and copyright even though they weren’t explicitly defined:
In fact, you can return a default config even if you don’t have a settings file. This is to support certain nbdev commands work outside of nbdev repos:
You can customise nbdev for all repositories for your user with a ~/.config/nbdev/settings.ini file (by default, although we follow the broader XDG specification). For example, you could globally disable nbdev’s Jupyter hooks by creating a user settings file with jupyter_hooks = False.
Add or update __version__ in the main __init__.py of the library.
+
Python modules require a __init.py__ file in all directories that are modules. We assume that all directories containing a python file (including in subdirectories of any depth) is a module, and therefore add a __init__.py to each.
+
+
with tempfile.TemporaryDirectory() as d:
+ d = Path(d)
+ (d/'a/b').mkdir(parents=True)
+ (d/'a/b/f.py').touch()
+ (d/'a/c').mkdir()
+ add_init(d)
+assertnot (d/'a/c'/_init).exists(), "Should not add init to dir without py file"
+for e in [d, d/'a', d/'a/b']: assert (e/_init).exists(),f"Missing init in {e}"
procs names the optional processors you wish to run on the exported cells of your notebook.
+
N.B.: the black_format processor is passed in by default. But it is a no-op, unless black_formatting=True is set in your settings.ini configuration. You can omit it from nbdev_export on the command line by passing in --procs.
Symbol names are taken from libraries registered using the ‘nbdev’ entry point. By default, all libraries with this entry point are searched, but full symbol names (including module prefix) are required.
md ="""This is a link to `numpy.array` and to `get_config` but not a link to `foobar`.
+And not a link to <code>dict2nb</code>.
+
+ This is not a link to `get_config`
+
+```
+This isn't a link to `get_config` either
+```"""
+
+
+
print(NbdevLookup('nbdev').linkify(md))
+
+
This is a link to [`numpy.array`](https://numpy.org/doc/stable/reference/generated/numpy.array.html#numpy.array) and to [`get_config`](https://nbdev.fast.ai/api/config.html#get_config) but not a link to `foobar`.
+And not a link to <code>dict2nb</code>.
+
+ This is not a link to `get_config`
+
+```
+This isn't a link to `get_config` either
+```
Specify dest where the module(s) will be exported to, and optionally a class to use to create the module (ModuleMaker, by default).
+
Exported cells are stored in a dict called modules, where the keys are the modules exported to. Those without an explicit module are stored in the '#' key, which will be exported to default_exp.
Processor to remove cell magics from exported code
+
+
+
+
+
Details
+
+
+
+
+
cell
+
Cell to format
+
+
+
+
scrub_magics is a processor that scrubs the jupyter “magics” lines out of exported cells. This can be helpful when using tools like sparkmagic or just Jupyter’s builtin magics in an nbdev project.
+
Usage:
+This behavior can be enabled by passing scrub_magics into the --procs flag of the nbdev_export command. - nbdev_export --procs scrub_magics - nbdev_export --procs 'scrub_magics black_format'
+
Example:
+
A cell like below could export the line "hello nbdev" into the bar module. And the %%spark magic line would be omitted.
+
%%spark
+#|export bar
+"hello nbdev"
+
It will export as something similar to this:
+
# %% ../path/to/01_bar.ipynb 1
+"hello nbdev"
+
+
_cell = read_nb('../../tests/export_procs.ipynb')['cells'][2]
+scrub_magics(_cell)
+test_eq(_cell.source, '''#|export bar
+"hello nbdev"''')
We’ll also check that our ‘everything’ file exports correctly:
+
+
nb_export(everything_fn, 'tmp')
+
+g = exec_new('import tmp.everything; from tmp.everything import *')
+_alls = L("a b d e m n o p q".split())
+for s in _alls.map("{}_y"): assert s in g, s
+for s in"c_y_nall _f_y_nall g_n h_n i_n j_n k_n l_n".split(): assert s notin g, s
+for s in _alls.map("{}_y") + ["c_y_nall", "_f_y_nall"]: asserthasattr(g['tmp'].everything,s), s
+
+
That notebook should also export one extra function to tmp.some.thing:
A YAML and formatted-markdown frontmatter processor
+
YAML frontmatter can be added to notebooks in one of two ways:
+
+
By adding a raw notebook cell with --- as the first and last lines, and YAML between them, or
+
A specially formatted markdown cell. The first line should be start with a single # (creating an H1 heading), and becomes the title. Then, optionally, a line beginning with > (creating a quote block), which becomes the description. Finally, zero or more lines beginning with - (creating a list), each of which contains YAML. (If you already have “title” defined in frontmatter in a raw cell, then markdown cells will be ignored.)
+
+
For instance, our test notebook contains the following markdown cell:
+
# a title
+> A description
+- key1: value1
+- key2: value2
+- categories: [c1, c2]
+
It also contains the following raw cell:
+
---
+execute:
+ echo: false
+---
+
When we process with FrontmatterProc, these will both be removed, and a single raw cell will be added to the top, containing the combined YAML frontmatter:
+
+
+
+
+
\ No newline at end of file
diff --git a/api/images/release.svg b/api/images/release.svg
new file mode 100644
index 000000000..2fa08a3f9
--- /dev/null
+++ b/api/images/release.svg
@@ -0,0 +1 @@
+
\ No newline at end of file
diff --git a/api/images/token.png b/api/images/token.png
new file mode 100644
index 000000000..3560f3eb5
Binary files /dev/null and b/api/images/token.png differ
diff --git a/api/index.html b/api/index.html
new file mode 100644
index 000000000..13ce7a7a2
--- /dev/null
+++ b/api/index.html
@@ -0,0 +1,1087 @@
+
+
+
+
+
+
+
+
+
+API – nbdev
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
API
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
This section contains API details for each of nbdev’s python submodules. This reference documentation is mainly useful for people looking to customise or build on top of nbdev, or wanting detailed information about how nbdev works.
Helper class to create exported library from notebook source cells
+
In order to export a notebook, we need an way to create a Python file. ModuleMaker fills that role. Pass in the directory where you want to module created, the name of the module, the path of the notebook source, and set is_new to True if this is a new file being created (rather than an existing file being added to). The location of the saved module will be in fname. Finally, if the source in the notebooks should not be parsed by Python (such as partial class declarations in cells), parse should be set to False.
+
+
Note: If doing so, then the __all__ generation will be turned off as well.
+
+
+
mm = ModuleMaker(dest='tmp', name='test.testing', nb_path=Path.cwd()/'01_export.ipynb', is_new=True)
+mm.fname
Pass all_cells=[] or parse=False if you don’t want any __all__ added.
+
Passing parse=False is also handy for when writing broken up functions or classes that ast.parse might not like but still want it to be exported, such as having a cell with:
+
#|export
+class A:
+
Note that by doing so we cannot properly generate a __all__, so we assume that it is unwanted.
+
+
am = ModuleMaker(dest='tmp', name='test.testing_noall', nb_path=Path.cwd()/'01_export.ipynb', is_new=True, parse=False)
+am.fname
try:
+ g = exec_import('tmp.test.testing', '*')
+for s in"a c d".split(): assert s in g, s
+assert'b'notin g
+assert g['a']() isNone
+finally: shutil.rmtree('tmp')
When working with jupyter notebooks (which are json files behind the scenes) and GitHub, it is very common that a merge conflict (that will add new lines in the notebook source file) will break some notebooks you are working on. This module defines the function nbdev_fix to fix those notebooks for you, and attempt to automatically merge standard conflicts. The remaining ones will be delimited by markdown cells like this:
+
<<<<<< HEAD
+
+
# local code here
+
+
======
+
+
# remote code here
+
+
>>>>>> a7ec1b0bfb8e23b05fd0a2e6cafcb41cd0fb1c35
+
Below is an example of broken notebook. The json format is broken by the lines automatically added by git. Such a file can’t be opened in jupyter notebook.
Note that in this example, the second conflict is easily solved: it just concerns the execution count of the second cell and can be solved by choosing either option without really impacting your notebook. This is the kind of conflict we will fix automatically. The first conflict is more complicated as it spans across two cells and there is a cell present in one version, not the other. Such a conflict (and generally the ones where the inputs of the cells change form one version to the other) aren’t automatically fixed, but we will return a proper json file where the annotations introduced by git will be placed in markdown cells.
+
+
+
Creating a merged notebook
+
The approach we use is to first “unpatch” the conflicted file, regenerating the two files it was originally created from. Then we redo the diff process, but using cells instead of text lines.
Takes a string with conflict markers and returns the two original files, and their branch names
+
The result of “unpatching” our conflicted test notebook is the two original notebooks it would have been created from. Each of these original notebooks will contain valid JSON:
Create working notebook from conflicted notebook nbname
+
+
+
+
+
+
+
+
+
+
+
Type
+
Default
+
Details
+
+
+
+
+
nbname
+
str
+
+
Notebook filename to fix
+
+
+
outname
+
str
+
None
+
Filename of output notebook (defaults to nbname)
+
+
+
nobackup
+
bool_arg
+
True
+
Do not backup nbname to nbname.bak if outname not provided
+
+
+
theirs
+
bool
+
False
+
Use their outputs and metadata instead of ours
+
+
+
noprint
+
bool
+
False
+
Do not print info about whether conflicts are found
+
+
+
+
This begins by optionally backing the notebook fname to fname.bak in case something goes wrong. Then it parses the broken json, solving conflicts in cells. Every conflict that only involves metadata or outputs of cells will be solved automatically by using the local (theirs==False) or the remote (theirs==True) branch. Otherwise, or for conflicts involving the inputs of cells, the json will be repaired by including the two version of the conflicted cell(s) with markdown cells indicating the conflicts. You will be able to open the notebook again and search for the conflicts (look for <<<<<<<) then fix them as you wish.
+
A message will be printed indicating whether the notebook was fully merged or if conflicts remain.
This implements a git merge driver for notebooks that automatically resolves conflicting metadata and outputs, and splits remaining conflicts as separate cells so that the notebook can be viewed and fixed in Jupyter. The easiest way to install it is by running nbdev_install_hooks.
+
This works by first running Git’s default merge driver, and then nbdev_fix if there are still conflicts. You can set nbdev_fix’s theirs argument using the THEIRS environment variable, for example:
# "fastcore: An Underrated Python Library"
+
+> A unique python library that extends the python programming language and provides utilities that enhance productivity.
+- author: "<a href='https://twitter.com/HamelHusain'>Hamel Husain</a>"
+- toc: false
+- image: images/copied_from_nb/fastcore_imgs/td.png
+- comments: true
+- search_exclude: true
+- hide: true
+- categories: [fastcore, fastai]
+- permalink: /fastcore/
+- badges: true
+
+
+
After migrating the notebook, the front matter is moved to a raw cell, and some of the fields are converted to be compliant with Quarto. Furthermore, aliases may be added in order to prevent broken links:
---
+
+toc: true
+layout: post
+description: A minimal example of using markdown with fastpages.
+categories: [markdown]
+title: An Example Markdown Post
+
+
+---
+
+# Example Markdown Post
---
+aliases:
+- /markdown/2020/01/14/test-markdown-post
+categories:
+- markdown
+date: '2020-01-14'
+description: A minimal example of using markdown with fastpages.
+layout: post
+title: An Example Markdown Post
+toc: true
+
+---
+
+# Example Markdown Post
+
+## Basic setup
+
+Jekyll requires blog post files to b
+
+
+
+
#hide
+_res = fp_md_fm('../../tests/2022-09-06-homeschooling.md')
+test_eq(_res,
+"""---
+aliases:
+- /2022/09/06/homeschooling
+author: Rachel Thomas
+categories:
+- advice
+- health
+date: '2022-09-06'
+description: You can permanently damage your back, neck, and wrists from working without
+ an ergonomic setup. Learn how to create one for less at home.
+image: /images/ergonomic1-short.jpg
+summary: You can permanently damage your back, neck, and wrists from working without
+ an ergonomic setup. Learn how to create one for less at home.
+tags: advice health
+title: 'Essential Work-From-Home Advice: Cheap and Easy Ergonomic Setups'
+
+---
+
+Lorem ipsum
+""")
+
+
+
+
Convert nbdev v1 projects to nbdev v2
+
+
Directives
+
nbdev v2 directives start with a #| whereas v1 directives were comments without a pipe |.
for example, if any of the lines below are valid nbdev v1 directives, they replaced with a #|, but only before the first line of code:
+
+
+
+
Callouts
+
In fastpages, there was a markdown shortuct for callouts for Note, Tip, Important and Warning with block quotes (these only worked in notebooks). Since Quarto has its own callout blocks with markdown syntax, we do not implement these shortcuts in nbdev. Instead, we offer a manual conversion utility for these callouts so that you can migrate from fastpages to Quarto.
_callouts="""
+## Boxes / Callouts
+
+> Warning: There will be no second warning!
+
+Other text
+
+> Important: Pay attention! It's important.
+
+> Tip: This is my tip.
+
+> Note: Take note of `this.`
+"""
+
+
Gets converted to:
+
+
+
+## Boxes / Callouts
+
+:::{.callout-warning}
+
+There will be no second warning!
+
+:::
+
+Other text
+
+:::{.callout-important}
+
+Pay attention! It's important.
+
+:::
+
+
+
+
+
+
Video Embeds
+
In fastpages, you could embed videos with a simple markdown shortcut involving a block quote with the prefix youtube:, that looked like this
+
> youtube: https://youtu.be/XfoYk_Z5AkI
+
However, in Quarto you can use the video extension to embed videos.
Special comments at the start of a cell can be used to provide information to nbdev about how to process a cell, so we need to be able to find the location of these comments.
Take leading comment directives from lines of code in ss, remove #|, and split
+
Comment directives start with #|, followed by whitespace delimited tokens, which extract_directives extracts from the start of a cell, up until a blank line or a line containing something other than comments. The extracted lines are removed from the source.
+
+
exp = AttrDict(source ="""#|export module
+#|eval:false
+#| hide
+# | foo bar
+# |woo: baz
+1+2
+#bar""")
+
+# this one has #|hide: with a colon at the end, wich is quarto compliant
+exp2 = AttrDict(source ="""#|export module
+#|eval:false
+#| hide:
+# | foo bar
+# |woo: baz
+1+2
+#bar""")
+
+_answer = {'export':['module'], 'hide':[], 'eval:': ['false'], 'foo': ['bar'], 'woo:': ['baz']}
+
+test_eq(extract_directives(exp), _answer)
+test_eq(extract_directives(exp2), _answer)
+test_eq(exp.source, '#|eval: false\n# |woo: baz\n1+2\n#bar')
Cell processors can be callables (e.g regular functions), in which case they are called for every cell (set a cell’s source to None to remove the cell):
However, a more convenient way to handle comment directives is to use a class as a processor, and include a method in your class with the same name as your directive, surrounded by underscores:
+
+
class _PrintExample:
+def _printme_(self, cell, to_print): print(to_print)
+
+NBProcessor(everything_fn, _PrintExample()).process()
+
+
testing
+
+
+
In the case that your processor supports just one comment directive, you can just use a regular function, with the same name as your directive, but with an underscore appended – here printme_ is identical to _PrintExample above:
For more complex behavior, inherit from Processor, and override one of more of begin() (called before any cells are processed), cell() (called for each cell), and end() (called after all cells are processed). You can also include comment directives (such as the _printme example above) in these subclasses. Subclasses will automatically have access to self.nb, containing the processed notebook.
+
+
class CountCellProcessor(Processor):
+def begin(self):
+print(f"First cell:\n{self.nb.cells[0].source}")
+self.count=0
+def cell(self, cell):
+if cell.cell_type=='code': self.count +=1
+def end(self): print(f"* There were {self.count} code cells")
Insert Autogenerated Warning Into Notebook after the first cell.
+
This preprocessor inserts a warning in the markdown destination that the file is autogenerated. This warning is inserted in the second cell so we do not interfere with front matter.
+
+
res = _run_procs(insert_warning)
+assert"<!-- WARNING: THIS FILE WAS AUTOGENERATED!"in res
res = _run_procs([populate_language, add_show_docs])
+assert"show_doc(some_func)'"in res
+assert"show_doc(and_another)'"in res
+assert"show_doc(another_func)'"notin res
res = _run_procs(add_links)
+assert"[`numpy.array`](https://numpy.org/doc/stable/reference/generated/numpy.array.html#numpy.array)"in res
+assert"[`ModuleMaker`](https://nbdev.fast.ai/api/maker.html#modulemaker) but not a link to `foobar`."in res
+assert"A link in a docstring: [`ModuleMaker`](https://nbdev.fast.ai/api/maker.html#modulemaker)."in res
+assert"And not a link to <code>dict2nb</code>."in res
+ Auto-generated tagged releases and release notes from GitHub issues
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Overview
+
nbdev.release provides 3 commands that you can run from your shell to manage your changelog file and git releases:
+
+
nbdev_changelog: creates a CHANGELOG.md file from closed and labeled GitHub issues
+
nbdev_release_git: tags and creates a release in GitHub for the current version
+
nbdev_release_gh: calls nbdev_changelog, lets you edit the result, then pushes to git and calls nbdev_release_git
+
+
It provides 3 futher commands for releasing packages on pypi or conda:
+
+
nbdev_pypi: Create and upload a pypi installer
+
nbdev_conda: Create and upload a conda installer
+
nbdev_release_both: Create and upload both pypi and conda installers
+
+
Here’s a brief demonstration of how to use the changelog and git release tools in nbdev.release. This demo first creates an issue using the gh command line tool, and then closes it using git; you can also use GitHub’s web interface for both of these tasks. (Note that this functionality used to be in a project called fastrelease, so in the video the command line tools have different names, starting with fastrelease_ instead of nbdev_).
+
+
+
Setup
+
You’ll need to get a GitHub personal access token if you haven’t already. To do so, click here and enter “nbdev” in the “Note” section, and click the repo checkbox.
+
Then click “Generate Token” at the bottom of the screen, and copy the token (the long string of letters and numbers shown). You can easily do that by clicking the little clipboard icon next to the token.
+
+
It’s easiest to save the token as an environment variable GITHUB_TOKEN that can be automatically accessed. We recommend you do this is by adding the following to the end of your .bashrc or .zshrc file:
+
exportGITHUB_TOKEN=xxx
+
…and then replace the xxx with the token you just copied. It will automatically be avaialble whenever you start a new shell (but don’t forget to source that file or open a new shell after you change it.).
+
+
+
Creating release notes
+
Now you’re ready to create your release notes. These are created in a file called CHANGELOG.md. Here’s an example of what it creates: nbdev CHANGELOG.
+
All issues with the label bug, enhancement, or breaking that have been closed in your repo since your last release will be added to the top of this file. If you haven’t made any releases before, then all issues with those labels will be included.
+
Therefore, before you create or update CHANGELOG.md, go to your GitHub issues page, remove is:open from the filter, and label any issues you want included with one of the labels above. When you’ve done that, you can create or update your release notes by running in your terminal:
+
nbdev_changelog
+
The titles and bodies of each issue will be added. Open CHANGELOG.md in your editor and make any edits that you want, and then commit the file to your repo (remember to git add it!)
+
+
+
Tagging a release
+
You should now tag a release. This will create a tag in GitHub with your current version number in settings.ini, and will then make it into a release, using your latest release notes as the description of the release:
+
nbdev_release_git
+
After you run this, be sure to increment your version number in settings.ini. You can either edit it manually, or if you use nbdev it can be done for you by running:
+
nbdev_bump_version
+
+
+
Doing both (creating release notes, and tagging a release)
+
To complete both of the steps above, run:
+
nbdev_release_gh
+
See the screencast above for a demonstration of this.
To create a markdown changelog, first create a Release object, optionally passing a mapping from GitHub labels to markdown titles. Put your github token in a file named token at the root of your repo. Release attempts to fetch values for arguments from the following locations if not supplied:
+
+
owner: fetched from the field user in settings.ini. This is the owner name of the repository on GitHub. For example for the repo fastai/fastcore the owner would be fastai.
+
repo: fetched from the field lib_name in settings.ini. This is the name of the repository on GitHub. For example for the repo fastai/fastcore the owner would be fastcore.
+
token: fetched from a file named token at the root of your repo. Creating a token is discussed in the setup section.
+
groups: (optional) fetched from the field label_groups in settings.ini, which is a JSON string. This is a mapping from label names to titles in your release notes. If not specified, this defaults to:
All relevant pull requests and issues are fetched from the GitHub API, and are categorized according to a user-supplied mapping from labels to markdown headings.
Add --debug to the conda build command to debug any problems that occur. Note that the build step takes a few minutes. Add -u {org_name} to the anaconda upload command if you wish to upload to an organization, or pass upload_user to nbdev_conda_package.
+
NB: you need to first of all upload your package to PyPi, before creating the conda package.
DocmentTbl can render a markdown table showing docments if appropriate. This is an example of how a docments table will render for a function:
+
+
def _f(a, # description of param a
+ b=True, # description of param b
+ c:str=None
+ ) ->int: ...
+
+_dm = DocmentTbl(_f)
+_dm
+
+
+
+
+
+
Type
+
Default
+
Details
+
+
+
+
+
a
+
+
+
description of param a
+
+
+
b
+
bool
+
True
+
description of param b
+
+
+
c
+
str
+
None
+
+
+
+
Returns
+
int
+
+
+
+
+
+
+
+
If one column in the table has no information, for example because there are no default values, that column will not be shown. In the below example, the Default column, will not be shown. Additionally, if the return of the function is not annotated the Returns row will not be rendered:
+
+
def _f(a,
+ b, #param b
+ c #param c
+ ): ...
+
+_dm2 = DocmentTbl(_f)
+_dm2
+
+
+
+
+
+
Details
+
+
+
+
+
a
+
+
+
+
b
+
param b
+
+
+
c
+
param c
+
+
+
+
+
+
DocmentTbl also works on classes. By default, the __init__ will be rendered:
+
+
class _Test:
+def__init__(self,
+ a, # description of param a
+ b=True, # description of param b
+ c:str=None):
+ ...
+
+def foo(self,
+ c:int, # description of param c
+ d=True, # description of param d
+ ):
+ ...
+
+
+
DocmentTbl(_Test)
+
+
+
+
+
+
Type
+
Default
+
Details
+
+
+
+
+
a
+
+
+
description of param a
+
+
+
b
+
bool
+
True
+
description of param b
+
+
+
c
+
str
+
None
+
+
+
+
+
+
+
You can also pass a method to be rendered as well:
+
+
DocmentTbl(_Test.foo)
+
+
+
+
+
+
Type
+
Default
+
Details
+
+
+
+
+
c
+
int
+
+
description of param c
+
+
+
d
+
bool
+
True
+
description of param d
+
+
+
+
+
+
+
+
+
Documentation For An Object
+
Render the signature as well as the docments to show complete documentation for an object.
You can use show_doc to document apis of functions, classes or methods.
+
+
+
Numpy Docstrings
+
if you have numpy docstrings instead of docments, show_doc will attempt to parse and render those just like docments.
+
+
+
+
+
+
+Warning
+
+
+
+
Numpy docstring formatting is very strict. If your docstrings do not strictly adhere to the numpy format, it will not be parsed properly and information about parameters and return values may not properly be rendered in the table below the signature. Where possible, we recommend using docments to annonate your function instead.
+
+
+
+
+
+
show_doc on Classes
+
show_doc works on Classes, too, including when you use @patch.
+
You can define methods for the class Foo with @patch which is convenient in allowing you to break up code for documentation in notebooks.
+
Class properties also work with showdoc.
+
+
+
Pluggable renderers
+
You can replace the default markdown show_doc renderer with custom renderers. For instance, nbdev comes with a simple example for rendering with raw HTML.
+ Propagate small changes in the library back to notebooks
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
The library is primarily developed in notebooks so any big changes should be made there. But sometimes, it’s easier to fix small bugs or typos in the modules directly. nbdev_update is the function that will propagate those changes back to the corresponding notebooks. Note that you can’t create new cells or reorder cells with that functionality, so your corrections should remain limited.
Sometimes you may wish to override one or more of the skip_flags, in which case you can use the argument force_flags which will remove the appropriate tag(s) from skip_flags. This is useful because skip_flags are meant to be set in the tst_flags field of settings.ini, whereas force_flags are usually passed in by the user.
+
+
+
+
+
\ No newline at end of file
diff --git a/blog/index.xml b/blog/index.xml
new file mode 100644
index 000000000..95484c20b
--- /dev/null
+++ b/blog/index.xml
@@ -0,0 +1,782 @@
+
+
+
+nbdev
+https://nbdev.fast.ai/blog/
+
+Create delightful software with Jupyter Notebooks
+quarto-1.5.55
+Mon, 07 Nov 2022 00:00:00 GMT
+
+ Create A 🤗 Space From A Notebook
+ Hamel Husain
+ https://nbdev.fast.ai/blog/posts/2022-11-07-spaces/
+
+
+
+Image created with Stable Diffusion from this space
+
+
+
Hugging Face Spaces provides an easy ways to deploy a web app with python. Gradio is one of my favorite tools for building these web apps. For example, the cover-image for this blog post was generated with a Gradio App!1 Gradio also allows you to prototype your web apps in notebooks which allows you to iterate fast. Unfortunately, Hugging Face Spaces requires you to package your web application code as a python script named app.py.
+
However, thanks to nbdev, you can deploy a Gradio app to Spaces from a notebook without having to refactor your code into a script! When you finish this tutorial, you will have an app that looks like this:
+
+
+
+A Gradio app that shows the size of a HF Dataset.
+
+
Authoring your spaces in notebooks offers a number of benefits such as the ability to:
+
+
Document your code (with quarto or nbdev)
+
Prototype and author your code (with nbdev.export.export_nb)
+
Test your code (with nbdev_test)
+
+
… All from the same environment!
+
+
1. Create a Gradio-enabled Space on Hugging Face
+
The first step is to create a space and select the appropriate sdk (which is Gradio in this example), according to these instructions:
+
+
+
+create a Hugging Face Space
+
+
+
After you are done creating the space, clone the repo locally. In this example, I ran the command git clone https://huggingface.co/spaces/hamel/hfspace_demo.
+
+
+
2. Create A Notebook
+
Before getting started you will want to install the dependencies for this tutorial:
Create a notebook called app.ipynb in the root of your newly cloned repo. Alternatively, download the notebook and follow along.
+
+
+
+
+
+
+Download the notebook and follow along
+
+
+
+
This blog post is a verbose version of the “notebook” you can use to create a Gradio app. However, it can be useful to see just the code without any of the prose. A concise version of this notebook is here. I recommend taking a look at this notebook during or after you read this blog post.
+
+
+
+
+
3. Make an app with Gradio
+
Below, I will create a gradio app in a notebook and show you how to deploy it to Hugging Face Spaces.
+
First, lets import the libraries we need, which in this case are gradio and fastcore:
+
#|export
+import gradio as gr
+from fastcore.net import urljson, HTTPError
+
Next, write the functions your gradio app will use. Because of nbdev, you can prototype and package your code all in one place. The special comment #|export marks which cells will be sent to a python script (more on this later). Note that there are only three cells in this notebook with the #|export directive.
+
#|export
+def size(repo:str):
+"Returns the size in GB of a HuggingFace Dataset."
+ url =f'https://huggingface.co/api/datasets/{repo}'
+try: resp = urljson(f'{url}/treesize/main')
+except HTTPError: returnf'Did not find repo: {url}'
+ gb = resp['size'] /1e9
+returnf'{gb:.2f} GB'
+
size takes as an input a Hugging Face Dataset repo and returns the total size in GB of the data.
You can construct a simple UI with the gradio.interface and then call the launch method of that interface to display a preview in a notebook. This is a great way to test your app to see if it works:
Running on local URL: http://127.0.0.1:7861
+
+To create a public link, set `share=True` in `launch()`.
+
+
+
Note how running the launch() method in a notebook runs a webserver in the background. Below, I call the close() method to close the webserver.
+
# this is only necessary in a notebook
+iface.close()
+
+
Closing server running on port: 7861
+
+
+
+
4. Convert This Notebook Into A Gradio App
+
In order to host this code on Hugging Face Spaces, you will export parts of this notebook to a script named app.py. As a reminder, this is what the special #|export comment that you have seen in cells above do! You can export code from this notebook like so:
+
from nbdev.export import nb_export
+nb_export('app.ipynb', lib_path='.', name='app')
+
+
Understanding what is generated
+
Notice how the contents of app.py only contain the exported cells from this notebook:
+
!cat app.py
+
+
# AUTOGENERATED! DO NOT EDIT! File to edit: app.ipynb.
+
+# %% auto 0
+__all__ = ['iface', 'size']
+
+# %% app.ipynb 6
+import gradio as gr
+from fastcore.net import urljson, HTTPError
+
+# %% app.ipynb 8
+def size(repo:str):
+"Returns the size in GB of a HuggingFace Dataset."
+ url =f'https://huggingface.co/api/datasets/{repo}'
+try: resp = urljson(f'{url}/treesize/main')
+except HTTPError: returnf'Did not find repo: {url}'
+ gb = resp['size'] /1e9
+returnf'{gb:.2f} GB'
+
+# %% app.ipynb 12
+iface = gr.Interface(fn=size, inputs=gr.Text(value="tglcourse/CelebA-faces-cropped-128"), outputs="text")
+iface.launch(height=450, width=500)
+
+
+
+
Fill out requirements.txt
+
You must supply a requirements.txt file so the Gradio app knows how to build your dependencies. In this example, the only dependency other than Gradio is fastcore. You don’t need to specify Gradio itself as a dependency in requirements.txt, so our requirements.txt file has only one dependency:
+
%%writefile requirements.txt
+fastcore
+
Writing requirements.txt
+
Note: you may want to add operating system dependencies in addition to python dependencies. You can do this via a packages.txt file as documented here.
+
+
+
+
5. Launch Your Gradio App
+
To launch your gradio app, you need to commit the changes to the Hugging Face repo:
Hopefully you felt something magical while doing this example. Even though the target application required you to write a python script (app.py), you didn’t have to refactor your script from a notebook! I believe that you shouldn’t have to refactor your code and switch contexts even when creating python packages! If this intrigues you, check out nbdev
+
+
+
+
+
+
Footnotes
+
+
+
The prompt that generated the cover image is: “A data scientist at a computer in a futuristic city with a view of the planet Jupyter in the night sky, trending on artstation, high detail, science-fiction”↩︎
+
+
]]>
+ https://nbdev.fast.ai/blog/posts/2022-11-07-spaces/
+ Mon, 07 Nov 2022 00:00:00 GMT
+
+
+
+ The Jupyter+git problem is now solved
+ Jeremy Howard
+ https://nbdev.fast.ai/blog/posts/2022-08-25-jupyter-git/
+ Originally posted on the fast.ai blog
+
+
Jupyter notebooks don’t work with git by default. With nbdev2, the Jupyter+git problem has been totally solved. It provides a set of hooks which provide clean git diffs, solve most git conflicts automatically, and ensure that any remaining conflicts can be resolved entirely within the standard Jupyter notebook environment. To get started, follow the directions on Git-friendly Jupyter.
+
+
+
The Jupyter+git problem
+
Jupyter notebooks are a powerful tool for scientists, engineers, technical writers, students, teachers, and more. They provide an ideal notebook environment for interactively exploring data and code, writing programs, and documenting the results as dashboards, books, or blogs.
+
But when collaborating with others, this ideal environment goes up in smoke. That’s because tools such as git, which are the most popular approaches for asynchronous collaboration, makes notebooks unusable. Literally. Here’s what it looks like if you and a colleague both modify a notebook cell (including, in many cases, simply executing a cell withuout changing it), and then try to open that notebook later:
+
+
+
+What merge conflicts normally do to Jupyter Notebooks
+
+
+
The reason for this stems from a fundamental incompatibility between the format Jupyter notebooks use (JSON) and the format that git conflict markers assume by default (plain lines of text). This is what it looks like when git adds its conflict markers to a notebook:
That’s not valid JSON, and therefore Jupyter can’t open it. Conflicts are particularly common in notebooks, because Jupyter changes the following every time you run a notebook:
+
+
Every cell includes a number indicating what order it was run in. If you and a colleague run the cells in different orders, you’ll have a conflict in every single cell! This would take a very long time to fix manually
+
For every figure, such as a plot, Jupyter includes not only the image itself in the notebook, but also a plain text description that includes the id (like a memory address) of the object, such as <matplotlib.axes._subplots.AxesSubplot at 0x7fbc113dbe90>. This changes every time you execute a notebook, and therefore will create a conflict every time two people execute this cell
+
Some outputs may be non-deterministic, such as a notebook that uses random numbers, or that interacts with a service that provides different outputs over time (such as a weather service)
+
Jupyter adds metadata to the notebook describing the environment it was last run in, such as the name of the kernel. This often varies across installations, and therefore two people saving a notebook (even without and other changes) will often end up with a conflict in the metadata.
+
+
All these changes to notebook files also make git diffs of notebooks very verbose. This can make code reviews a challenge, and make git repos more bulky than necessary.
+
The result of these problems is that many Jupyter users feel that collaborating with notebooks is a clunky, error-prone, and frustrating experience. (We’ve even seen people on social media describe Jupyter’s notebook format as “stupid” or “terrible”, despite otherwise professing their love for the software!)
+
It turns out, however, that Jupyter and git can work together extremely well, with none of the above problems at all. All that’s needed is a bit of special software…
+
+
+
The solution
+
Jupyter and git are both well-designed software systems that provide many powerful extensibility mechanisms. It turns out that we can use these to fully and automatically solve the Jupyter+git problem. We identified two categories of problems in the previous section:
+
+
git conflicts lead to broken notebooks
+
Unnecessary conflicts due to metadata and outputs.
+
+
In our newly released nbdev2, an open source Jupyter-based development platform, we’ve solve each of the problems:
+
+
A new merge driver for git provides “notebook-native” conflict markers, resulting in notebooks that can be opened directly in Jupyter, even when there are git conflicts
+
A new save hook for Jupyter automatically removes all unnecessary metadata and non-deterministic cell output.
+
+
Here’s what a conflict looks like in Jupyter with nbdev’s merge driver:
+
+
+
+
As you see, the local and remote change are each clearly displayed as separate cells in the notebook, allowing you to simply delete the version you don’t want to keep, or combine the two cells as needed.
+
The techniques used to make the merge driver work are quite fascinating – let’s dive into the details!
+
+
The nbdev2 git merge driver
+
We provide here a summary of the git merge driver – for full details and source code see the nbdev.merge docs. Amazingly enough, the entire implementation is just 58 lines of code!
+
The basic idea is to first “undo” the original git merge which created the conflict, and then “redo” it at a cell level (instead of a line level) and looking only at cell source (not outputs or metadata). The “undoing” is straightforward: just create two copies of the conflicted file (representing the local and remove versions of the file), go through each git conflict marker, and replace the conflict section with either the local or remote version of the code.
+
Now that we’ve got the original local and remote notebooks, we can load the json using execnb.nbio, which will then give us an array of cells for each notebook. Now we’re up to the interesting bit – creating cell-level diffs based only on the cell source.
+
The Python standard library contains a very flexible and effective implementation of a diff algorithm in the difflib module. In particular, the SequenceMatcher class provides the fundamental building blocks for implementing your own conflict resolution system. We pass the two sets of cells (remote and local) to SequenceMatcher(...).get_matching_blocks(), and it returns a list of each section of cells that match (i.e. have no conflicts/differences). We can then go through each matching section and copy them into the final notebook, and through each non-matching section and copy in each of the remote and local cells (add cells between them to mark the conflicts).
+
Making SequenceMatcher work with notebook cells (represented in nbdev by the NbCell class) requires only adding __hash__ and __eq__ methods to NbCell. In each case, these methods are defined to look only at the actual source code, and not at any metadata or outputs. As a result, SequenceMatcher will only show differences in source code, and will ignore differences in everything else.
+
With a single line of configuration, we can ask git to call our python script, instead of its default line-based implementation, any time it is merging changes. nbdev_install_hooks sets up this configuration automatically, so after running it, git conflicts become much less common, and never result in broken notebooks.
+
+
+
The nbdev2 Jupyter save hook
+
Solving git merges locally is extremely helpful, but we need to solve them remotely as well. For instance, if a contributor submits a pull request (PR), and then someone else commits to the same notebook before the PR is merged, the PR might now have a conflict like this:
This conflict shows that the two contributors have run cells in different orders (or perhaps one added a couple of cells above in the notebook), so their commits have conflicting execution counts. GitHub will refuse to allow this PR to be merged until this conflict is fixed.
+
But of course we don’t really care about the conflict at all – it doesn’t matter what, if any, execution count is stored in the notebook. So we’d really prefer to ignore this difference entirely!
+
Thankfully, Jupyter provides a “pre-save” hook which allows code to be run every time a notebook is saved. nbdev uses this to set up a hook which removes all unnecessary metadata (including execution_count) on saving. That means there’s no pointless conflicts like the one above, because no commits will have this information stored in the first place.
+
+
+
+
Background
+
Here at fast.ai we use Jupyter for everything. All our tests, documentation, and module source code for all of our many libraries is entirely developed in notebooks (using nbdev, of course!) And we use git for all our libraries too. Some of our repositories have many hundreds of contributors. Therefore solving the Jupyter+git problem has been critical for us. The solution presented here is the result of years of work by many people.
+
Our first approach, developed by Stas Bekman and me, was to use git “smudge” and “clean” filters that automatically rewrote all notebook json to remove unneeded metadata when committing. This helped a bit, but git quite often ended up in an odd state where it was impossible to merge.
+
In nbdev v1 Sylvain Gugger created an amazing tool called nbdev_fix_merge which used very clever custom logic to manually fix merge conflicts in notebooks, to ensure that they could opened in Jupyter. For nbdev v2 I did a from-scratch rewrite of every part of the library, and I realised that we could replace the custom logic with the SequenceMatcher approach described above.
+
None of these steps fully resolved the Jupyter+git problem, since we were getting frequent merge errors caused by the smudge/clean git filters, and conflicts required manually running nbdev_fix_merge. Wasim Lorgat realised that we could resolve the smudge/clean issue by moving that logic into an nbdev save hook, and avoid the manual fix step by moving that logic into a git merge driver. This resolved the final remaining issues! (I was actually quite stunned that Wasim went from our first discussion of the outstanding problems, to figuring out how to solve all of them, in the space of about two days…)
+
+
+
The result
+
The new tools in nbdev2, which we’ve been using internally for the last few months, have been transformational to our workflow. The Jupyter+git problem has been totally solved. I’ve seen no unnecessary conflicts, cell-level merges have worked like magic, and on the few occassions where I’ve changed the source in the same cell as a collaborator, fixing the conflict in Jupyter has been straightforward and convenient.
+
+
+
Postscript: other Jupyter+git tools
+
+
ReviewNB
+
There is one other tool which we’ve found very helpful in using Jupyter with git, which is ReviewNB. ReviewNB solves the problem of doing pull requests with notebooks. GitHub’s code review GUI only works well for line-based file formats, such as plain python scripts. This works fine with the Python modules that nbdev exports, and I often do reviews directly on the Python files, instead of the source notebooks.
+
However, much of the time I’d rather do reviews on the source notebooks, because:
+
+
I want to review the documentation and tests, not just the implementation
+
I want to see the changes to cell outputs, such as charts and tables, not just the code.
+
+
For this purpose, ReviewNB is perfect. Just like nbdev makes git merges and commits Jupyter-friendly, ReviewNB makes code reviews Jupyter-friendly. A picture is worth a thousand words, so rather than trying to explain, I’ll just show this picture from the ReviewNB website of what PRs look like in their interface:
+
+
+
+
An alternative solution: Jupytext
+
Another potential solution to the Jupyter+git problem might be to use Jupytext. Jupytext saves notebooks in a line-based format, instead of in JSON. This means that all the usual git machinery, such as merges and PRs, works fine. Jupytext can even use Quarto’s format, qmd, as a format for saving notebooks, which then can be used to generate a website.
+
Jupytext can be a bit tricky to manage when you want to save your cell outputs (which I generally want to do, since many of my notebooks take a long time to run – e.g training deep learning models.) Whilst Jupytext can save outputs in a linked ipynb file, managing this linkage gets complex, and ends up with the Jupyter+git problem all over again! If you don’t need to save outputs, then you might find Jupytext sufficient – although of course you’ll miss out on the cell-based code reviews of ReviewNB and your users won’t be able to read your notebooks properly when they’re browsing GitHub.
+
+
+
nbdime
+
There’s also an interesting project called nbdime which has its own git drivers and filters. Since they’re not really compatible with nbdev (partly because they tackle some of the same problems in different ways) I haven’t used them much, so haven’t got an informed opinion about them. However I do use nbdime’s Jupyter extension sometimes, which provides a view similar to ReviewNB, but for local changes instead of PRs.
+
If you want to try to yourself, follow the directions on Git-friendly Jupyter to get started.
+
+
+
+
+
+ ]]>
+ https://nbdev.fast.ai/blog/posts/2022-08-25-jupyter-git/
+ Thu, 25 Aug 2022 00:00:00 GMT
+
+
+
+ nbdev+Quarto: A new secret weapon for productivity
+ Hamel Husain
+ Jeremy Howard
+ https://nbdev.fast.ai/blog/posts/2022-07-28-nbdev2/
+ Originally posted on the fast.ai blog
+
+
+
+A single notebook can create a python module, tests, CI, pypi/conda packages, and more.
+
+
+
+
Our new secret weapon for productivity
+
Today we’re excited to announce that we’ve teamed up with Quarto to give nbdev superpowers. nbdev offers Python programmers a common set of tools for using Jupyter notebooks to:
+
+
Write & distribute software packages
+
Test code, and
+
Author documentation and technical articles
+
+
Although notebooks are already widely used for once-off exploratory work, it’s less well-known that they are perfectly capable of writing quality software. In fact, we’ve used nbdev for a wide range of software projects over the last three years, including deep learning libraries,API clients, Python language extensions, terminal user interfaces, and more. We discovered that it is not only capable of writing great software but that it has also increased our productivity by 300% or more. With nbdev, developers simply write notebooks with lightweight markup and get high-quality documentation, tests, continuous integration, and packaging for free! Nbdev has allowed us to maintain and scale many open source projects. Pull requests are often accompanied by detailed documentation and tests–contributors simply write notebooks.
+
This is why we’re excited to share nbdev v2. It’s rewritten from the ground up, with much-anticipated features including:
+
+
Interoperation with non-nbdev codebases for tasks like documentation
+
Support for any static site generator
+
Wide variety of output mediums such as blogs, papers, slides, and websites
+
A faster Jupyter kernel, which also means faster tests
+
Cleaner and more extensible API, which supports custom directives, custom module exporters, and more
+
+
+
+
nbdev in industry
+
We have piloted nbdev at several companies. We were delighted to receive the following feedback, which fits our own experience using and developing nbdev:
+
+
+
+
+
+
+
David Berg, on using nbdev for internal documentation at Netflix: “Prior to using nbdev, documentation was the most cumbersome aspect of our software development process… Using nbdev allows us to spend more time creating rich prose around the many code snippets guaranteeing the whole experience is robust. nbdev has turned what was once a chore into a natural extension of the notebook-based testing we were already doing.”
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Erik Gaasedelen, on using nbdev in production at Lyft: “I use this in production at my company. It’s an awesome tool… nbdev streamlines everything so I can write docs, tests, and code all in one place… The packaging is also really well thought out. From my point of view it is close to a Pareto improvement over traditional Python library development.”
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Hugo Bowne-Anderson, on using nbdev for Outerbounds: “nbdev has transformed the way we write documentation. Gone are the days of worrying about broken code examples when our API changes or [due to] human errors associated with copying & pasting code into markdown files. The authoring experience of nbdev… [allows] us to write prose and live code in a unified interface, which allows more experimentation… On top of this, nbdev allows us to include unit tests in our documentation which mitigates the burden of maintaining the docs over time.”
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Roxanna Pourzand, on using nbdev for Transform: “We’re so excited about using nbdev. Our product is technical so our resulting documentation includes a lot of code-based examples. Before nbdev, we had no way of maintaining our code examples and ensuring that it was up-to-date for both command inputs and outputs. It was all manual. With nbdev, we now have this under control in a sustainable way. Since we’ve deployed these docs, we also had a situation where we were able to identify a bug in one of our interfaces, which we found by seeing the error that was output in the documentation.”
+
+
+
+
+
+
+
+
+
What’s nbdev?
+
Nbdev embraces the dynamic nature of python and REPL-driven development in ways that traditional IDEs and software development workflows cannot. We thoroughly discussed the motivation, history, and goals of nbdev in this initial launch post three years ago. The creator of Jupyter, Fernando Pérez, told us:
+
+
[Nbdev] should be celebrated and used a lot more - I have kept a tab with your original nbdev blog post open for months in Chrome because of how often I refer to it and point others to this work
+
+
In short, nbdev embraces ideas from literate programming and exploratory programming. These paradigms have been revisited in platforms like XCode Playgrounds and languages like Smalltalk, LISP, and Mathematica. With nbdev, we sought to push these paradigms even further by enabling it for one of the most popular dynamic programming languages in the world: Python.
Hamel: When I use nbdev, my colleagues are often astounded by how quickly I can create and distribute high-quality python packages. I consider nbdev to be a superpower that allows me to create tests and documentation without any additional friction, which makes all of my projects more maintainable. I also find writing software with nbdev to be more fun and productive as I can iterate very fast on ideas relative to more traditional software engineering workflows. Lastly, with nbdev I can also use traditional text-based IDEs if I want to, so I get the best of both worlds.
+
+
+
What we learned after three years of using nbdev
+
While nbdev was originally developed to simplify the software development workflow for various fast.ai projects, we found that users wanted to extend nbdev to:
+
+
Write and publish blog posts, books, papers, and other types of documents with Jupyter Notebooks
+
Document existing codebases not written in nbdev
+
Accommodate traditional Python conventions–for those constrained in how their code is organized and formatted
+
Publish content using any static site generator
+
+
While we created projects such as fastpages and fastdoc to accomplish some of these tasks, we realized that it would be better to have a single set of flexible tools to accomplish all of them. To this end, we were extremely excited to discover Quarto, an open-source technical publishing system built on pandoc.
+
Hamel: The more I used nbdev for creating Python modules, the more I wanted to use it for writing blogs and documenting existing codebases. The ability to customize the way notebooks are rendered (hiding vs. showing cells, stripping output, etc.), along with the facilities for including unit tests, made it my go-to authoring tool for all technical content. I’m excited that nbdev2 unlocks all of these possibilities for everyone!
+
+
+
Enter Quarto: A pandoc super-processor
+
Quarto is a project that enables technical publishing with support for Jupyter Notebook, VSCode, Observable, and plaintext editors. Furthermore, Quarto enables the publishing of high-quality articles, reports, websites, and blogs in HTML, PDF, ePub, PowerPoint slides, and more. Quarto is maintained by RStudio, a company with a long history of products supporting literate programming, such as RMarkdown and RStudio.
+
Quarto is built on top of Pandoc, a universal document converter that supports nearly any format you can think of. Pandoc achieves this seemingly magical feat by representing documents in a common abstract syntax tree (AST) that serves as the medium through which different formats can be translated. By extension, Quarto allows you to generate content in almost any format you wish! You can use pandoc filters to modify the AST and the output format, which allows you to use any static site generator you want, and programmatically modify and generate content.
We also find Quarto compelling because user interfaces such as comment directives (comments that start with #|) correlate with nbdev. In fact, we even learned that nbdev inspired Quarto in this regard! In general, Quarto and nbdev share many goals, and the Quarto team has been incredibly responsive to our suggestions. For example, the ability to create notebook filters to modify notebooks before rendering. Below is a screenshot of a Jupyter notebook rendered with Quarto and nbdev.
+
+
+
+
+Quarto rendering a Jupyter notebook written with nbdev
+
+
+
+
Finally, Quarto supports more programming languages than just Python and has been adding new features and fixing bugs at an impressive speed. This gives us confidence that we will be able to expand nbdev to support more use cases in the future. We discuss some of these future directions in the closing section.
+
+
+
A blazing fast notebook kernel: execnb
+
A core component of nbdev is executing and testing notebooks programmatically. It is important that this notebook runner executes with minimal overhead to maintain our goal of providing a delightful developer experience. This is why we built execnb, a lightweight notebook runner for Python kernels, which executes notebooks blazingly fast. Furthermore, execnb allows parameterized execution of notebooks.
+
Hamel: I have been an enthusiastic user of tools like papermill that programmatically run notebooks for use-cases like creating dashboards or enabling new kinds of machine learning workflows. I believe execnb unlocks even more possibilities with its ability to inject arbitrary code at any place in a notebook, as well as the ability to pass callbacks that run before and/or after cells are executed. This opens up possibilities to create new types of workflows with notebooks that I am excited about exploring in the near future.
+
+
+
Towards a dialect of python that embraces its dynamic nature
+
One way to understand nbdev is part of an ecosystem that is designed to embrace Python’s dynamic properties for REPL-driven software engineering. Similar to Clojure, our goal is to provide tools that remove all friction from using the REPL in your programming workflow. We believe that the REPL enhances developer workflows thanks to context-sensitive auto-completion, signature inspection, and documentation–all based on the actual state of your code, and none of which are available in IDEs that depend solely on static analysis. We have found that for this reason, nbdev, with its Jupyter notebook foundation, makes programming significantly more productive and enjoyable.
+
Our efforts to support REPL-driven development and literate programming are not limited to nbdev. We maintain a number of libraries that extend python to bolster this programming experience. The most notable of these libraries is fastcore, which extends Python in terms of testing, documenting code, metaprogramming, attribute helpers, enhanced representations of objects, and notebook-friendly patching. This blog post offers a gentle introduction to fastcore. In addition to literate programming, fastcore encourages conventions such as brevity and efficient use of vertical space so you can accomplish more with significantly less code. For example, below is a simple decorator that enables notebook-friendly patching:
+
+
+
+@patch decorator from fastcore
+
+
+
We believe that this combination of a new developer workflow (nbdev), Python extensions (fastcore), and associated norms form a new dialect of Python that is centered on leveraging its dynamic nature–in contrast to an ever-growing trend toward static analysis. We suspect that this dialect of Python will be more productive for programmers in many scenarios. We are framing this ecosystem as a “dialect” as it is still very much Python and is approachable by anyone who is familiar with the language. Furthermore, despite nbdev’s notebook workflow, our tools generate plaintext modules that can be navigated and edited with text-based IDEs, allowing programmers to experience the best of both worlds, if they desire.
+
Hamel: I believe this framing of a Python dialect is key to properly understanding what nbdev is. While it may be tempting to get stuck on specific features or technical details of nbdev, it is useful to zoom out to understand the overall intent of creating a better workflow rather than conforming too rigidly to existing ones. A good analogy is TypeScript’s relationship with JavaScript: it is an extension of an existing programming language that supports a new way of programming. I encourage you to treat nbdev in a similar fashion: be willing to try new ways of programming and observe which tradeoffs resonate with you. At the very least, I believe nbdev is a fun way to experience a different way of writing software, which will broaden your horizons about programming in general, all without having to learn an entirely new programming language!
+
+
+
The future of nbdev
+
While we are excited about nbdev2, we believe we have only scratched the surface of what’s possible. We are considering the following features:
+
+
Supporting more languages beyond Python, such as Julia, R and JavaScript
+
Offering interfaces for executing parameterized notebooks that mimic Python scripts
+
Extensions for more static site generators and filters
+
Supporting alternate testing backends, such as pytest
+
Supporting a greater number of docstring formats, such as Google-style docstrings
+
More options to use plain-text or human readable notebook backends other than JSON
+
+
If you have interesting ideas about how nbdev can be extended, please drop and chat with us on discord or post a message in the forums.
+
+
+
How you can get started with nbdev
+
Our project’s website is at nbdev.fast.ai, where we will be posting tutorials, examples, and more documentation in the coming days.
+
+
+
Thank You
+
This new version of nbdev was a team effort by many wonderful people. We want to highlight two people who have made outstanding contributions:
+
+
Wasim Lorgat was instrumental across different areas, including significant contributions to fastcore, execnb, and nbdev, as well as the implementation of the new nbdev home page. With Wasim’s help, we were able to push nbdev to a new level of functionality and quality.
+
JJ Allaire is not only the CEO of RStudio but also the steward of Quarto. JJ was incredibly responsive and eager to work with us on nbdev and added many features to Quarto specifically with nbdev in mind, such as notebook filters. We were also astounded by the attention to detail and the pace at which bugs are addressed. This new version of nbdev would not have been possible without JJ’s help, and we are excited to continue to work with him.
To celebrate the launch of nbdev v2 and Quarto, Jeremy sat down with the CEO of Posit (previously known as RStudio, the company behind Quarto), JJ Allaire, to talk about software development, scientific publishing, R, Python, literate programming, and much more.
+A single notebook can create a python module, tests, CI, pypi/conda packages, and more.
+
+
+
+
Our new secret weapon for productivity
+
Today we’re excited to announce that we’ve teamed up with Quarto to give nbdev superpowers. nbdev offers Python programmers a common set of tools for using Jupyter notebooks to:
+
+
Write & distribute software packages
+
Test code, and
+
Author documentation and technical articles
+
+
Although notebooks are already widely used for once-off exploratory work, it’s less well-known that they are perfectly capable of writing quality software. In fact, we’ve used nbdev for a wide range of software projects over the last three years, including deep learning libraries,API clients, Python language extensions, terminal user interfaces, and more. We discovered that it is not only capable of writing great software but that it has also increased our productivity by 300% or more. With nbdev, developers simply write notebooks with lightweight markup and get high-quality documentation, tests, continuous integration, and packaging for free! Nbdev has allowed us to maintain and scale many open source projects. Pull requests are often accompanied by detailed documentation and tests–contributors simply write notebooks.
+
This is why we’re excited to share nbdev v2. It’s rewritten from the ground up, with much-anticipated features including:
+
+
Interoperation with non-nbdev codebases for tasks like documentation
+
Support for any static site generator
+
Wide variety of output mediums such as blogs, papers, slides, and websites
+
A faster Jupyter kernel, which also means faster tests
+
Cleaner and more extensible API, which supports custom directives, custom module exporters, and more
+
+
+
+
nbdev in industry
+
We have piloted nbdev at several companies. We were delighted to receive the following feedback, which fits our own experience using and developing nbdev:
+
+
+
+
+
+
+
David Berg, on using nbdev for internal documentation at Netflix: “Prior to using nbdev, documentation was the most cumbersome aspect of our software development process… Using nbdev allows us to spend more time creating rich prose around the many code snippets guaranteeing the whole experience is robust. nbdev has turned what was once a chore into a natural extension of the notebook-based testing we were already doing.”
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Erik Gaasedelen, on using nbdev in production at Lyft: “I use this in production at my company. It’s an awesome tool… nbdev streamlines everything so I can write docs, tests, and code all in one place… The packaging is also really well thought out. From my point of view it is close to a Pareto improvement over traditional Python library development.”
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Hugo Bowne-Anderson, on using nbdev for Outerbounds: “nbdev has transformed the way we write documentation. Gone are the days of worrying about broken code examples when our API changes or [due to] human errors associated with copying & pasting code into markdown files. The authoring experience of nbdev… [allows] us to write prose and live code in a unified interface, which allows more experimentation… On top of this, nbdev allows us to include unit tests in our documentation which mitigates the burden of maintaining the docs over time.”
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Roxanna Pourzand, on using nbdev for Transform: “We’re so excited about using nbdev. Our product is technical so our resulting documentation includes a lot of code-based examples. Before nbdev, we had no way of maintaining our code examples and ensuring that it was up-to-date for both command inputs and outputs. It was all manual. With nbdev, we now have this under control in a sustainable way. Since we’ve deployed these docs, we also had a situation where we were able to identify a bug in one of our interfaces, which we found by seeing the error that was output in the documentation.”
+
+
+
+
+
+
+
+
+
What’s nbdev?
+
Nbdev embraces the dynamic nature of python and REPL-driven development in ways that traditional IDEs and software development workflows cannot. We thoroughly discussed the motivation, history, and goals of nbdev in this initial launch post three years ago. The creator of Jupyter, Fernando Pérez, told us:
+
+
[Nbdev] should be celebrated and used a lot more - I have kept a tab with your original nbdev blog post open for months in Chrome because of how often I refer to it and point others to this work
+
+
In short, nbdev embraces ideas from literate programming and exploratory programming. These paradigms have been revisited in platforms like XCode Playgrounds and languages like Smalltalk, LISP, and Mathematica. With nbdev, we sought to push these paradigms even further by enabling it for one of the most popular dynamic programming languages in the world: Python.
Hamel: When I use nbdev, my colleagues are often astounded by how quickly I can create and distribute high-quality python packages. I consider nbdev to be a superpower that allows me to create tests and documentation without any additional friction, which makes all of my projects more maintainable. I also find writing software with nbdev to be more fun and productive as I can iterate very fast on ideas relative to more traditional software engineering workflows. Lastly, with nbdev I can also use traditional text-based IDEs if I want to, so I get the best of both worlds.
+
+
+
What we learned after three years of using nbdev
+
While nbdev was originally developed to simplify the software development workflow for various fast.ai projects, we found that users wanted to extend nbdev to:
+
+
Write and publish blog posts, books, papers, and other types of documents with Jupyter Notebooks
+
Document existing codebases not written in nbdev
+
Accommodate traditional Python conventions–for those constrained in how their code is organized and formatted
+
Publish content using any static site generator
+
+
While we created projects such as fastpages and fastdoc to accomplish some of these tasks, we realized that it would be better to have a single set of flexible tools to accomplish all of them. To this end, we were extremely excited to discover Quarto, an open-source technical publishing system built on pandoc.
+
Hamel: The more I used nbdev for creating Python modules, the more I wanted to use it for writing blogs and documenting existing codebases. The ability to customize the way notebooks are rendered (hiding vs. showing cells, stripping output, etc.), along with the facilities for including unit tests, made it my go-to authoring tool for all technical content. I’m excited that nbdev2 unlocks all of these possibilities for everyone!
+
+
+
Enter Quarto: A pandoc super-processor
+
Quarto is a project that enables technical publishing with support for Jupyter Notebook, VSCode, Observable, and plaintext editors. Furthermore, Quarto enables the publishing of high-quality articles, reports, websites, and blogs in HTML, PDF, ePub, PowerPoint slides, and more. Quarto is maintained by RStudio, a company with a long history of products supporting literate programming, such as RMarkdown and RStudio.
+
Quarto is built on top of Pandoc, a universal document converter that supports nearly any format you can think of. Pandoc achieves this seemingly magical feat by representing documents in a common abstract syntax tree (AST) that serves as the medium through which different formats can be translated. By extension, Quarto allows you to generate content in almost any format you wish! You can use pandoc filters to modify the AST and the output format, which allows you to use any static site generator you want, and programmatically modify and generate content.
We also find Quarto compelling because user interfaces such as comment directives (comments that start with #|) correlate with nbdev. In fact, we even learned that nbdev inspired Quarto in this regard! In general, Quarto and nbdev share many goals, and the Quarto team has been incredibly responsive to our suggestions. For example, the ability to create notebook filters to modify notebooks before rendering. Below is a screenshot of a Jupyter notebook rendered with Quarto and nbdev.
+
+
+
+
+Quarto rendering a Jupyter notebook written with nbdev
+
+
+
+
Finally, Quarto supports more programming languages than just Python and has been adding new features and fixing bugs at an impressive speed. This gives us confidence that we will be able to expand nbdev to support more use cases in the future. We discuss some of these future directions in the closing section.
+
+
+
A blazing fast notebook kernel: execnb
+
A core component of nbdev is executing and testing notebooks programmatically. It is important that this notebook runner executes with minimal overhead to maintain our goal of providing a delightful developer experience. This is why we built execnb, a lightweight notebook runner for Python kernels, which executes notebooks blazingly fast. Furthermore, execnb allows parameterized execution of notebooks.
+
Hamel: I have been an enthusiastic user of tools like papermill that programmatically run notebooks for use-cases like creating dashboards or enabling new kinds of machine learning workflows. I believe execnb unlocks even more possibilities with its ability to inject arbitrary code at any place in a notebook, as well as the ability to pass callbacks that run before and/or after cells are executed. This opens up possibilities to create new types of workflows with notebooks that I am excited about exploring in the near future.
+
+
+
Towards a dialect of python that embraces its dynamic nature
+
One way to understand nbdev is part of an ecosystem that is designed to embrace Python’s dynamic properties for REPL-driven software engineering. Similar to Clojure, our goal is to provide tools that remove all friction from using the REPL in your programming workflow. We believe that the REPL enhances developer workflows thanks to context-sensitive auto-completion, signature inspection, and documentation–all based on the actual state of your code, and none of which are available in IDEs that depend solely on static analysis. We have found that for this reason, nbdev, with its Jupyter notebook foundation, makes programming significantly more productive and enjoyable.
+
Our efforts to support REPL-driven development and literate programming are not limited to nbdev. We maintain a number of libraries that extend python to bolster this programming experience. The most notable of these libraries is fastcore, which extends Python in terms of testing, documenting code, metaprogramming, attribute helpers, enhanced representations of objects, and notebook-friendly patching. This blog post offers a gentle introduction to fastcore. In addition to literate programming, fastcore encourages conventions such as brevity and efficient use of vertical space so you can accomplish more with significantly less code. For example, below is a simple decorator that enables notebook-friendly patching:
+
+
+
+@patch decorator from fastcore
+
+
+
We believe that this combination of a new developer workflow (nbdev), Python extensions (fastcore), and associated norms form a new dialect of Python that is centered on leveraging its dynamic nature–in contrast to an ever-growing trend toward static analysis. We suspect that this dialect of Python will be more productive for programmers in many scenarios. We are framing this ecosystem as a “dialect” as it is still very much Python and is approachable by anyone who is familiar with the language. Furthermore, despite nbdev’s notebook workflow, our tools generate plaintext modules that can be navigated and edited with text-based IDEs, allowing programmers to experience the best of both worlds, if they desire.
+
Hamel: I believe this framing of a Python dialect is key to properly understanding what nbdev is. While it may be tempting to get stuck on specific features or technical details of nbdev, it is useful to zoom out to understand the overall intent of creating a better workflow rather than conforming too rigidly to existing ones. A good analogy is TypeScript’s relationship with JavaScript: it is an extension of an existing programming language that supports a new way of programming. I encourage you to treat nbdev in a similar fashion: be willing to try new ways of programming and observe which tradeoffs resonate with you. At the very least, I believe nbdev is a fun way to experience a different way of writing software, which will broaden your horizons about programming in general, all without having to learn an entirely new programming language!
+
+
+
The future of nbdev
+
While we are excited about nbdev2, we believe we have only scratched the surface of what’s possible. We are considering the following features:
+
+
Supporting more languages beyond Python, such as Julia, R and JavaScript
+
Offering interfaces for executing parameterized notebooks that mimic Python scripts
+
Extensions for more static site generators and filters
+
Supporting alternate testing backends, such as pytest
+
Supporting a greater number of docstring formats, such as Google-style docstrings
+
More options to use plain-text or human readable notebook backends other than JSON
+
+
If you have interesting ideas about how nbdev can be extended, please drop and chat with us on discord or post a message in the forums.
+
+
+
How you can get started with nbdev
+
Our project’s website is at nbdev.fast.ai, where we will be posting tutorials, examples, and more documentation in the coming days.
+
+
+
Thank You
+
This new version of nbdev was a team effort by many wonderful people. We want to highlight two people who have made outstanding contributions:
+
+
Wasim Lorgat was instrumental across different areas, including significant contributions to fastcore, execnb, and nbdev, as well as the implementation of the new nbdev home page. With Wasim’s help, we were able to push nbdev to a new level of functionality and quality.
+
JJ Allaire is not only the CEO of RStudio but also the steward of Quarto. JJ was incredibly responsive and eager to work with us on nbdev and added many features to Quarto specifically with nbdev in mind, such as notebook filters. We were also astounded by the attention to detail and the pace at which bugs are addressed. This new version of nbdev would not have been possible without JJ’s help, and we are excited to continue to work with him.
To celebrate the launch of nbdev v2 and Quarto, Jeremy sat down with the CEO of Posit (previously known as RStudio, the company behind Quarto), JJ Allaire, to talk about software development, scientific publishing, R, Python, literate programming, and much more.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/blog/posts/2022-07-28-nbdev2/lang_rank.png b/blog/posts/2022-07-28-nbdev2/lang_rank.png
new file mode 100644
index 000000000..28826a51e
Binary files /dev/null and b/blog/posts/2022-07-28-nbdev2/lang_rank.png differ
diff --git a/blog/posts/2022-07-28-nbdev2/logo_lyft.png b/blog/posts/2022-07-28-nbdev2/logo_lyft.png
new file mode 100644
index 000000000..170875541
Binary files /dev/null and b/blog/posts/2022-07-28-nbdev2/logo_lyft.png differ
diff --git a/blog/posts/2022-07-28-nbdev2/logo_netflix.png b/blog/posts/2022-07-28-nbdev2/logo_netflix.png
new file mode 100644
index 000000000..4659eda8a
Binary files /dev/null and b/blog/posts/2022-07-28-nbdev2/logo_netflix.png differ
diff --git a/blog/posts/2022-07-28-nbdev2/logo_ob.png b/blog/posts/2022-07-28-nbdev2/logo_ob.png
new file mode 100644
index 000000000..c0fd56a6e
Binary files /dev/null and b/blog/posts/2022-07-28-nbdev2/logo_ob.png differ
diff --git a/blog/posts/2022-07-28-nbdev2/logo_transform.png b/blog/posts/2022-07-28-nbdev2/logo_transform.png
new file mode 100644
index 000000000..f61024963
Binary files /dev/null and b/blog/posts/2022-07-28-nbdev2/logo_transform.png differ
diff --git a/blog/posts/2022-07-28-nbdev2/nb_quarto.png b/blog/posts/2022-07-28-nbdev2/nb_quarto.png
new file mode 100644
index 000000000..f58906959
Binary files /dev/null and b/blog/posts/2022-07-28-nbdev2/nb_quarto.png differ
diff --git a/blog/posts/2022-07-28-nbdev2/patch.png b/blog/posts/2022-07-28-nbdev2/patch.png
new file mode 100644
index 000000000..444ebce04
Binary files /dev/null and b/blog/posts/2022-07-28-nbdev2/patch.png differ
diff --git a/blog/posts/2022-08-25-jupyter-git/friendly-conflict.png b/blog/posts/2022-08-25-jupyter-git/friendly-conflict.png
new file mode 100644
index 000000000..8a8ab6ae8
Binary files /dev/null and b/blog/posts/2022-08-25-jupyter-git/friendly-conflict.png differ
diff --git a/blog/posts/2022-08-25-jupyter-git/index.html b/blog/posts/2022-08-25-jupyter-git/index.html
new file mode 100644
index 000000000..5019b4b8c
--- /dev/null
+++ b/blog/posts/2022-08-25-jupyter-git/index.html
@@ -0,0 +1,1024 @@
+
+
+
+
+
+
+
+
+
+
+
+
+The Jupyter+git problem is now solved – nbdev
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
The Jupyter+git problem is now solved
+
+
+
+
+ Previously, using git with Jupyter could create conflicts and break notebooks. With nbdev2, the problem has been totally solved.
+
Jupyter notebooks don’t work with git by default. With nbdev2, the Jupyter+git problem has been totally solved. It provides a set of hooks which provide clean git diffs, solve most git conflicts automatically, and ensure that any remaining conflicts can be resolved entirely within the standard Jupyter notebook environment. To get started, follow the directions on Git-friendly Jupyter.
+
+
+
The Jupyter+git problem
+
Jupyter notebooks are a powerful tool for scientists, engineers, technical writers, students, teachers, and more. They provide an ideal notebook environment for interactively exploring data and code, writing programs, and documenting the results as dashboards, books, or blogs.
+
But when collaborating with others, this ideal environment goes up in smoke. That’s because tools such as git, which are the most popular approaches for asynchronous collaboration, makes notebooks unusable. Literally. Here’s what it looks like if you and a colleague both modify a notebook cell (including, in many cases, simply executing a cell withuout changing it), and then try to open that notebook later:
+
+
+
+What merge conflicts normally do to Jupyter Notebooks
+
+
+
The reason for this stems from a fundamental incompatibility between the format Jupyter notebooks use (JSON) and the format that git conflict markers assume by default (plain lines of text). This is what it looks like when git adds its conflict markers to a notebook:
That’s not valid JSON, and therefore Jupyter can’t open it. Conflicts are particularly common in notebooks, because Jupyter changes the following every time you run a notebook:
+
+
Every cell includes a number indicating what order it was run in. If you and a colleague run the cells in different orders, you’ll have a conflict in every single cell! This would take a very long time to fix manually
+
For every figure, such as a plot, Jupyter includes not only the image itself in the notebook, but also a plain text description that includes the id (like a memory address) of the object, such as <matplotlib.axes._subplots.AxesSubplot at 0x7fbc113dbe90>. This changes every time you execute a notebook, and therefore will create a conflict every time two people execute this cell
+
Some outputs may be non-deterministic, such as a notebook that uses random numbers, or that interacts with a service that provides different outputs over time (such as a weather service)
+
Jupyter adds metadata to the notebook describing the environment it was last run in, such as the name of the kernel. This often varies across installations, and therefore two people saving a notebook (even without and other changes) will often end up with a conflict in the metadata.
+
+
All these changes to notebook files also make git diffs of notebooks very verbose. This can make code reviews a challenge, and make git repos more bulky than necessary.
+
The result of these problems is that many Jupyter users feel that collaborating with notebooks is a clunky, error-prone, and frustrating experience. (We’ve even seen people on social media describe Jupyter’s notebook format as “stupid” or “terrible”, despite otherwise professing their love for the software!)
+
It turns out, however, that Jupyter and git can work together extremely well, with none of the above problems at all. All that’s needed is a bit of special software…
+
+
+
The solution
+
Jupyter and git are both well-designed software systems that provide many powerful extensibility mechanisms. It turns out that we can use these to fully and automatically solve the Jupyter+git problem. We identified two categories of problems in the previous section:
+
+
git conflicts lead to broken notebooks
+
Unnecessary conflicts due to metadata and outputs.
+
+
In our newly released nbdev2, an open source Jupyter-based development platform, we’ve solve each of the problems:
+
+
A new merge driver for git provides “notebook-native” conflict markers, resulting in notebooks that can be opened directly in Jupyter, even when there are git conflicts
+
A new save hook for Jupyter automatically removes all unnecessary metadata and non-deterministic cell output.
+
+
Here’s what a conflict looks like in Jupyter with nbdev’s merge driver:
+
+
+
+
As you see, the local and remote change are each clearly displayed as separate cells in the notebook, allowing you to simply delete the version you don’t want to keep, or combine the two cells as needed.
+
The techniques used to make the merge driver work are quite fascinating – let’s dive into the details!
+
+
The nbdev2 git merge driver
+
We provide here a summary of the git merge driver – for full details and source code see the nbdev.merge docs. Amazingly enough, the entire implementation is just 58 lines of code!
+
The basic idea is to first “undo” the original git merge which created the conflict, and then “redo” it at a cell level (instead of a line level) and looking only at cell source (not outputs or metadata). The “undoing” is straightforward: just create two copies of the conflicted file (representing the local and remove versions of the file), go through each git conflict marker, and replace the conflict section with either the local or remote version of the code.
+
Now that we’ve got the original local and remote notebooks, we can load the json using execnb.nbio, which will then give us an array of cells for each notebook. Now we’re up to the interesting bit – creating cell-level diffs based only on the cell source.
+
The Python standard library contains a very flexible and effective implementation of a diff algorithm in the difflib module. In particular, the SequenceMatcher class provides the fundamental building blocks for implementing your own conflict resolution system. We pass the two sets of cells (remote and local) to SequenceMatcher(...).get_matching_blocks(), and it returns a list of each section of cells that match (i.e. have no conflicts/differences). We can then go through each matching section and copy them into the final notebook, and through each non-matching section and copy in each of the remote and local cells (add cells between them to mark the conflicts).
+
Making SequenceMatcher work with notebook cells (represented in nbdev by the NbCell class) requires only adding __hash__ and __eq__ methods to NbCell. In each case, these methods are defined to look only at the actual source code, and not at any metadata or outputs. As a result, SequenceMatcher will only show differences in source code, and will ignore differences in everything else.
+
With a single line of configuration, we can ask git to call our python script, instead of its default line-based implementation, any time it is merging changes. nbdev_install_hooks sets up this configuration automatically, so after running it, git conflicts become much less common, and never result in broken notebooks.
+
+
+
The nbdev2 Jupyter save hook
+
Solving git merges locally is extremely helpful, but we need to solve them remotely as well. For instance, if a contributor submits a pull request (PR), and then someone else commits to the same notebook before the PR is merged, the PR might now have a conflict like this:
This conflict shows that the two contributors have run cells in different orders (or perhaps one added a couple of cells above in the notebook), so their commits have conflicting execution counts. GitHub will refuse to allow this PR to be merged until this conflict is fixed.
+
But of course we don’t really care about the conflict at all – it doesn’t matter what, if any, execution count is stored in the notebook. So we’d really prefer to ignore this difference entirely!
+
Thankfully, Jupyter provides a “pre-save” hook which allows code to be run every time a notebook is saved. nbdev uses this to set up a hook which removes all unnecessary metadata (including execution_count) on saving. That means there’s no pointless conflicts like the one above, because no commits will have this information stored in the first place.
+
+
+
+
Background
+
Here at fast.ai we use Jupyter for everything. All our tests, documentation, and module source code for all of our many libraries is entirely developed in notebooks (using nbdev, of course!) And we use git for all our libraries too. Some of our repositories have many hundreds of contributors. Therefore solving the Jupyter+git problem has been critical for us. The solution presented here is the result of years of work by many people.
+
Our first approach, developed by Stas Bekman and me, was to use git “smudge” and “clean” filters that automatically rewrote all notebook json to remove unneeded metadata when committing. This helped a bit, but git quite often ended up in an odd state where it was impossible to merge.
+
In nbdev v1 Sylvain Gugger created an amazing tool called nbdev_fix_merge which used very clever custom logic to manually fix merge conflicts in notebooks, to ensure that they could opened in Jupyter. For nbdev v2 I did a from-scratch rewrite of every part of the library, and I realised that we could replace the custom logic with the SequenceMatcher approach described above.
+
None of these steps fully resolved the Jupyter+git problem, since we were getting frequent merge errors caused by the smudge/clean git filters, and conflicts required manually running nbdev_fix_merge. Wasim Lorgat realised that we could resolve the smudge/clean issue by moving that logic into an nbdev save hook, and avoid the manual fix step by moving that logic into a git merge driver. This resolved the final remaining issues! (I was actually quite stunned that Wasim went from our first discussion of the outstanding problems, to figuring out how to solve all of them, in the space of about two days…)
+
+
+
The result
+
The new tools in nbdev2, which we’ve been using internally for the last few months, have been transformational to our workflow. The Jupyter+git problem has been totally solved. I’ve seen no unnecessary conflicts, cell-level merges have worked like magic, and on the few occassions where I’ve changed the source in the same cell as a collaborator, fixing the conflict in Jupyter has been straightforward and convenient.
+
+
+
Postscript: other Jupyter+git tools
+
+
ReviewNB
+
There is one other tool which we’ve found very helpful in using Jupyter with git, which is ReviewNB. ReviewNB solves the problem of doing pull requests with notebooks. GitHub’s code review GUI only works well for line-based file formats, such as plain python scripts. This works fine with the Python modules that nbdev exports, and I often do reviews directly on the Python files, instead of the source notebooks.
+
However, much of the time I’d rather do reviews on the source notebooks, because:
+
+
I want to review the documentation and tests, not just the implementation
+
I want to see the changes to cell outputs, such as charts and tables, not just the code.
+
+
For this purpose, ReviewNB is perfect. Just like nbdev makes git merges and commits Jupyter-friendly, ReviewNB makes code reviews Jupyter-friendly. A picture is worth a thousand words, so rather than trying to explain, I’ll just show this picture from the ReviewNB website of what PRs look like in their interface:
+
+
+
+
An alternative solution: Jupytext
+
Another potential solution to the Jupyter+git problem might be to use Jupytext. Jupytext saves notebooks in a line-based format, instead of in JSON. This means that all the usual git machinery, such as merges and PRs, works fine. Jupytext can even use Quarto’s format, qmd, as a format for saving notebooks, which then can be used to generate a website.
+
Jupytext can be a bit tricky to manage when you want to save your cell outputs (which I generally want to do, since many of my notebooks take a long time to run – e.g training deep learning models.) Whilst Jupytext can save outputs in a linked ipynb file, managing this linkage gets complex, and ends up with the Jupyter+git problem all over again! If you don’t need to save outputs, then you might find Jupytext sufficient – although of course you’ll miss out on the cell-based code reviews of ReviewNB and your users won’t be able to read your notebooks properly when they’re browsing GitHub.
+
+
+
nbdime
+
There’s also an interesting project called nbdime which has its own git drivers and filters. Since they’re not really compatible with nbdev (partly because they tackle some of the same problems in different ways) I haven’t used them much, so haven’t got an informed opinion about them. However I do use nbdime’s Jupyter extension sometimes, which provides a view similar to ReviewNB, but for local changes instead of PRs.
+
If you want to try to yourself, follow the directions on Git-friendly Jupyter to get started.
+
+
+
+
+
+
+
+
+
+
+
+
+
\ No newline at end of file
diff --git a/blog/posts/2022-08-25-jupyter-git/unreadable-notebook.png b/blog/posts/2022-08-25-jupyter-git/unreadable-notebook.png
new file mode 100644
index 000000000..c7f9cb0b2
Binary files /dev/null and b/blog/posts/2022-08-25-jupyter-git/unreadable-notebook.png differ
diff --git a/blog/posts/2022-11-07-spaces/blog_cover.jpeg b/blog/posts/2022-11-07-spaces/blog_cover.jpeg
new file mode 100644
index 000000000..348cce5ef
Binary files /dev/null and b/blog/posts/2022-11-07-spaces/blog_cover.jpeg differ
diff --git a/blog/posts/2022-11-07-spaces/create_space.png b/blog/posts/2022-11-07-spaces/create_space.png
new file mode 100644
index 000000000..68d62a798
Binary files /dev/null and b/blog/posts/2022-11-07-spaces/create_space.png differ
diff --git a/blog/posts/2022-11-07-spaces/final_app.png b/blog/posts/2022-11-07-spaces/final_app.png
new file mode 100644
index 000000000..2a3185fcd
Binary files /dev/null and b/blog/posts/2022-11-07-spaces/final_app.png differ
diff --git a/blog/posts/2022-11-07-spaces/gradio_local.png b/blog/posts/2022-11-07-spaces/gradio_local.png
new file mode 100644
index 000000000..649523419
Binary files /dev/null and b/blog/posts/2022-11-07-spaces/gradio_local.png differ
diff --git a/blog/posts/2022-11-07-spaces/index.html b/blog/posts/2022-11-07-spaces/index.html
new file mode 100644
index 000000000..e6dc9784f
--- /dev/null
+++ b/blog/posts/2022-11-07-spaces/index.html
@@ -0,0 +1,1087 @@
+
+
+
+
+
+
+
+
+
+
+
+
+Create A 🤗 Space From A Notebook – nbdev
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Create A 🤗 Space From A Notebook
+
+
+
+
+ A demo of using nbdev with Hugging Face Spaces & Gradio.
+
+
+
+
+
+
+
+
Author
+
+
Hamel Husain
+
+
+
+
+
Published
+
+
November 7, 2022
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Image created with Stable Diffusion from this space
+
+
+
Hugging Face Spaces provides an easy ways to deploy a web app with python. Gradio is one of my favorite tools for building these web apps. For example, the cover-image for this blog post was generated with a Gradio App!1 Gradio also allows you to prototype your web apps in notebooks which allows you to iterate fast. Unfortunately, Hugging Face Spaces requires you to package your web application code as a python script named app.py.
+
However, thanks to nbdev, you can deploy a Gradio app to Spaces from a notebook without having to refactor your code into a script! When you finish this tutorial, you will have an app that looks like this:
+
+
+
+A Gradio app that shows the size of a HF Dataset.
+
+
Authoring your spaces in notebooks offers a number of benefits such as the ability to:
+
+
Document your code (with quarto or nbdev)
+
Prototype and author your code (with nbdev.export.export_nb)
+
Test your code (with nbdev_test)
+
+
… All from the same environment!
+
+
1. Create a Gradio-enabled Space on Hugging Face
+
The first step is to create a space and select the appropriate sdk (which is Gradio in this example), according to these instructions:
+
+
+
+create a Hugging Face Space
+
+
+
After you are done creating the space, clone the repo locally. In this example, I ran the command git clone https://huggingface.co/spaces/hamel/hfspace_demo.
+
+
+
2. Create A Notebook
+
Before getting started you will want to install the dependencies for this tutorial:
Create a notebook called app.ipynb in the root of your newly cloned repo. Alternatively, download the notebook and follow along.
+
+
+
+
+
+
+Download the notebook and follow along
+
+
+
+
This blog post is a verbose version of the “notebook” you can use to create a Gradio app. However, it can be useful to see just the code without any of the prose. A concise version of this notebook is here. I recommend taking a look at this notebook during or after you read this blog post.
+
+
+
+
+
3. Make an app with Gradio
+
Below, I will create a gradio app in a notebook and show you how to deploy it to Hugging Face Spaces.
+
First, lets import the libraries we need, which in this case are gradio and fastcore:
+
#|export
+import gradio as gr
+from fastcore.net import urljson, HTTPError
+
Next, write the functions your gradio app will use. Because of nbdev, you can prototype and package your code all in one place. The special comment #|export marks which cells will be sent to a python script (more on this later). Note that there are only three cells in this notebook with the #|export directive.
+
#|export
+def size(repo:str):
+"Returns the size in GB of a HuggingFace Dataset."
+ url =f'https://huggingface.co/api/datasets/{repo}'
+try: resp = urljson(f'{url}/treesize/main')
+except HTTPError: returnf'Did not find repo: {url}'
+ gb = resp['size'] /1e9
+returnf'{gb:.2f} GB'
+
size takes as an input a Hugging Face Dataset repo and returns the total size in GB of the data.
You can construct a simple UI with the gradio.interface and then call the launch method of that interface to display a preview in a notebook. This is a great way to test your app to see if it works:
Running on local URL: http://127.0.0.1:7861
+
+To create a public link, set `share=True` in `launch()`.
+
+
+
Note how running the launch() method in a notebook runs a webserver in the background. Below, I call the close() method to close the webserver.
+
# this is only necessary in a notebook
+iface.close()
+
+
Closing server running on port: 7861
+
+
+
+
4. Convert This Notebook Into A Gradio App
+
In order to host this code on Hugging Face Spaces, you will export parts of this notebook to a script named app.py. As a reminder, this is what the special #|export comment that you have seen in cells above do! You can export code from this notebook like so:
+
from nbdev.export import nb_export
+nb_export('app.ipynb', lib_path='.', name='app')
+
+
Understanding what is generated
+
Notice how the contents of app.py only contain the exported cells from this notebook:
+
!cat app.py
+
+
# AUTOGENERATED! DO NOT EDIT! File to edit: app.ipynb.
+
+# %% auto 0
+__all__ = ['iface', 'size']
+
+# %% app.ipynb 6
+import gradio as gr
+from fastcore.net import urljson, HTTPError
+
+# %% app.ipynb 8
+def size(repo:str):
+"Returns the size in GB of a HuggingFace Dataset."
+ url =f'https://huggingface.co/api/datasets/{repo}'
+try: resp = urljson(f'{url}/treesize/main')
+except HTTPError: returnf'Did not find repo: {url}'
+ gb = resp['size'] /1e9
+returnf'{gb:.2f} GB'
+
+# %% app.ipynb 12
+iface = gr.Interface(fn=size, inputs=gr.Text(value="tglcourse/CelebA-faces-cropped-128"), outputs="text")
+iface.launch(height=450, width=500)
+
+
+
+
Fill out requirements.txt
+
You must supply a requirements.txt file so the Gradio app knows how to build your dependencies. In this example, the only dependency other than Gradio is fastcore. You don’t need to specify Gradio itself as a dependency in requirements.txt, so our requirements.txt file has only one dependency:
+
%%writefile requirements.txt
+fastcore
+
Writing requirements.txt
+
Note: you may want to add operating system dependencies in addition to python dependencies. You can do this via a packages.txt file as documented here.
+
+
+
+
5. Launch Your Gradio App
+
To launch your gradio app, you need to commit the changes to the Hugging Face repo:
Hopefully you felt something magical while doing this example. Even though the target application required you to write a python script (app.py), you didn’t have to refactor your script from a notebook! I believe that you shouldn’t have to refactor your code and switch contexts even when creating python packages! If this intrigues you, check out nbdev
+
+
+
+
+
+
Footnotes
+
+
+
The prompt that generated the cover image is: “A data scientist at a computer in a futuristic city with a view of the planet Jupyter in the night sky, trending on artstation, high detail, science-fiction”↩︎
All of nbdev’s configuration is done through a file called settings.ini which lives in the root of your repo. It’s in ConfigParser format. For example, here’s the first few lines of nbdev’s settings.ini file
+
+
+
[DEFAULT]
+lib_name = nbdev
+description = Create delightful software with Jupyter Notebooks
+copyright = 2020 onwards, Jeremy Howard
+keywords = nbdev fastai jupyter notebook export
+user = fastai
+author = Jeremy Howard and Hamel Husain
+author_email = j@fast.ai
+branch = master
+min_python = 3.7
+
+
+
You can create this file with nbdev_create_config (in which case you pass the settings manually), or with nbdev_new (which sets it up automatically for you from your repo settings). Here are all of nbdev’s settings (excluding the path and cfg_name parameters which decide where the config file is saved):
+
+
+
+
+
+
+
+
+
+
+
+
+
Type
+
Default
+
Details
+
+
+
+
+
repo
+
str
+
None
+
Repo name
+
+
+
branch
+
str
+
None
+
Repo default branch
+
+
+
user
+
str
+
None
+
Repo username
+
+
+
author
+
str
+
None
+
Package author’s name
+
+
+
author_email
+
str
+
None
+
Package author’s email address
+
+
+
description
+
str
+
None
+
Short summary of the package
+
+
+
path
+
str
+
.
+
Path to create config file
+
+
+
cfg_name
+
str
+
settings.ini
+
Name of config file to create
+
+
+
lib_name
+
str
+
%(repo)s
+
Package name
+
+
+
git_url
+
str
+
https://github.com/%(user)s/%(repo)s
+
Repo URL
+
+
+
custom_sidebar
+
bool_arg
+
False
+
Use a custom sidebar.yml?
+
+
+
nbs_path
+
Path
+
nbs
+
Path to notebooks
+
+
+
lib_path
+
Path
+
None
+
Path to package root (default: repo with - replaced by _)
+
+
+
doc_path
+
Path
+
_docs
+
Path to rendered docs
+
+
+
tst_flags
+
str
+
notest
+
Test flags
+
+
+
version
+
str
+
0.0.1
+
Version of this release
+
+
+
doc_host
+
str
+
https://%(user)s.github.io
+
Hostname for docs
+
+
+
doc_baseurl
+
str
+
/%(repo)s
+
Base URL for docs
+
+
+
keywords
+
str
+
nbdev jupyter notebook python
+
Package keywords
+
+
+
license
+
str
+
apache2
+
License for the package
+
+
+
copyright
+
str
+
None
+
Copyright for the package, defaults to ‘current_year onwards, author’
+
+
+
status
+
str
+
3
+
Development status PyPI classifier
+
+
+
min_python
+
str
+
3.7
+
Minimum Python version PyPI classifier
+
+
+
audience
+
str
+
Developers
+
Intended audience PyPI classifier
+
+
+
language
+
str
+
English
+
Language PyPI classifier
+
+
+
recursive
+
bool_arg
+
True
+
Include subfolders in notebook globs?
+
+
+
black_formatting
+
bool_arg
+
False
+
Format libraries with black?
+
+
+
readme_nb
+
str
+
index.ipynb
+
Notebook to export as repo readme
+
+
+
title
+
str
+
%(lib_name)s
+
Quarto website title
+
+
+
allowed_metadata_keys
+
str
+
+
Preserve the list of keys in the main notebook metadata
+
+
+
allowed_cell_metadata_keys
+
str
+
+
Preserve the list of keys in cell level metadata
+
+
+
jupyter_hooks
+
bool_arg
+
False
+
Run Jupyter hooks?
+
+
+
clean_ids
+
bool_arg
+
True
+
Remove ids from plaintext reprs?
+
+
+
clear_all
+
bool_arg
+
False
+
Remove all cell metadata and cell outputs?
+
+
+
cell_number
+
bool_arg
+
True
+
Add cell number to the exported file
+
+
+
put_version_in_init
+
bool_arg
+
True
+
Add the version to the main init.py in nbdev_export
+
+
+
+
+
+
You can customise nbdev for all repositories for your user with a ~/.config/nbdev/settings.ini file.
+
In order for Git actions to run smoothly, add requirements and dev_requirements with required packages in settings.ini.
+ A cheat sheet of directives available in nbdev.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Directives are special comments that are preceded by #| that control:
+
+
Cell visibility in rendered documentation
+
How source code is generated from notebook cells
+
Execution of cells for tests and docs
+
+
nbdev augments Quarto by providing additional directives than what are available in Quarto. All Quarto directives can be used in nbdev notebooks.
+
This cheat sheet lists all nbdev directives in addition to some Quarto directives we believe are important. We recommend consulting the quarto docs to see all of the directives available to you.
+
To clarify the origin of directives we use the following emojis:
+
+
📓 for nbdev-only directives .
+
🔵 for Quarto directives (which also work in nbdev).
+
+
+
Cell Visibility
+
The following directives control cell visibility in rendered documentation:
+
+
📓 #|hide
+
Hide cell input and output.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
The following will result in the contents of the cell and it’s output from being hidden:
+
#|hide
+print('you will not see this')
+
Note that using #|hide is equivalent to using the Quarto directive #|include: false:
+
#|include: false
+print('you will not see this')
+
See the quarto docs for more information about #|include.
+
+
+
+
+
+
🔵 #|echo: <true|false>
+
Toggle the visibility of code-cell inputs.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
#|echo: false
+print('you can see the output but not the code!')
+
which results in:
+
you can see the output but not the code!
+
+
+
+
+
+
🔵 #|output: <true|false|asis>
+
Setting this to false hides the output of a cell. Setting this to asis renders the output as raw markdown.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
The following cell will not display any output:
+
#|output: false
+1+1
+
The following cell with #|output: asis will produce the output hello fastai rendered as markdown instead of a string:
The #|code-fold directive allows you to collapse code cells. When set to true, the element is collapsed by default, when set to show show the element is shown by default.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
When you set #|code-fold: true, the input cell is collapsed:
+
+
+Code
+
print('this is')
+print('output')
+print('that takes')
+print('lots of vertical space')
+
+
+
this is
+output
+that takes
+lots of vertical space
+
+
+
When you set #|code-fold: show the input cell is shown but still in a collapsible element:
+
+
+Code
+
print('this is')
+print('output')
+print('that takes')
+print('lots of vertical space')
+
+
+
this is
+output
+that takes
+lots of vertical space
+
+
+
+
+
+
+
+
+
Generating Source Code
+
The following directives control how source code is exported from code cells.
+
+
📓 #|default_exp <name>
+
Names the module where cells with the #|export directive will be exported to by default.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
#| default_exp baz
+
+# In a new notebook cell:
+
+#| export
+def my_function(): pass
+
If our package is named: bitsnbytes then we can do:
+
from bitsnbytes.baz import my_function
+
You can define the package name by using lib_name in settings.ini.
+
+
+
+
+
+
📓 #|export
+
Exports the items in the cell into the generated module and documentation.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
#|export
+def say_hello(to:str# name of person to say hello to
+ ):
+"Say hello to somebody"
+returnf'Hello {to}!'
+
The above cell will get exported to the module specified by #|default_exp. These exports are automatically included in __all__ for the module. To learn how export without inclusion in __all__, see the #|exporti directive.
+
Furthermore, the documentation for this function will automatically be rendered like this:
+
+
+
say_hello
+
+
say_hello (to:str)
+
+
Say hello to somebody
+
+
+
+
+
Type
+
Details
+
+
+
+
+
to
+
str
+
name of person to say hello to
+
+
+
+
The docs are generated from this export using show_doc. See these docs for a detailed discussion of show_doc.
+
+
+
+
+
+
+
📓 #|exporti
+
An internal export. Not included in __all__ or the docs. Useful for a function that is called by other functions in this module but is not part of the public API.
+
Equivalently, you can prefix your function or method with _ e.g. def _private(): pass.
+
+
+
📓 #|exports
+
A source export. Like #|export but in addition to showing docs via showdoc.show_doc, it also shows the source code.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
#|exports
+def say_hello(to):
+"Say hello to somebody"
+returnf'Hello {to}!'
+
this will produce the following output:
+
+
def say_hello(to):
+"Say hello to somebody"
+returnf'Hello {to}!'
+
+
+
+
say_hello
+
+
say_hello (to)
+
+
Say hello to somebody
+
+
+
+
+
+
+
+
Cell Execution
+
The following directives allow you to control how cells are executed during docs rendering and testing.
+
+
📓 #|exec_doc
+
Ensures that a cell is executed each time before generating docs. When a cell does not have this annotation, it is run according to the default rules described here.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
+
datetime.datetime.now()
+
+
datetime.datetime(2022, 8, 18, 9, 1, 43, 907609)
+
+
+
However with the annotation:
+
#|exec_doc
+datetime.datetime.now()
+
we can see that the time has been updated:
+
+
datetime.datetime.now()
+
+
datetime.datetime(2024, 7, 30, 22, 41, 12, 13519)
+
+
+
+
+
+
+
+
🔵 #|eval: <true|false>
+
When set to false, the cell is ignored during testing.
+
+
+
+
+
+
+Example
+
+
+
+
+
+
#|eval: false
+raiseException("I'm not raised because I'm not run")
+
+
+
+
+
+
Cell execution when there is no directive
+
When a cell has no directives, cells are run by nbdev according to the behavior described here.
+ How nbdev renders a documentation website for your project.
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Concepts
+
+
Source Files
+
There are two mediums in which you can author documentation in nbdev:
+
+
Jupyter Notebooks
+
Quarto Markdown (.qmd)
+
+
For most cases, you will use Jupyter Notebooks. However, you may choose to author a document in Quarto Markdown if there is no code on that particular page. When in doubt, we recommend using notebooks as they are more versatile.
+
In the case of notebooks, nbdev pre-processes them to add, remove or change the content before passing it to Quarto. In some cases, nbdev even executes certain cells in order to render the documentation properly. The mechanics of this are discussed in the Notebook Processor section below.
+
For markdown files, Quarto renders these directly.
+
+
+
Notebook Processor
+
Nbdev does special pre-processing on notebooks based on the following:
+
+
Directives: Directives are special comments that allow you to perform operations on cells. For example, the comment #|hide allows you to hide cell inputs and outputs. You can read more about directives on this page. Directives that are unique to nbdev and not supported by Quarto are removed from the notebook before being passed to Quarto.
+
Front Matter: Front matter allows you to specify document-level options so you don’t have to repeat them on each cell. (Similarly, _quarto.yml allows you to specify project-level options.) You can read more about Quarto front-matter here.
+
+
The directives and front-matter are used by a Processing Pipeline to transform notebooks. Many of these pre-processing steps are defined in nbdev.processors, which are then run by nbdev.process.NBProcessor. Some of these pre-processing steps involve running code (with execnb) in order to dynamically render API documentation. This process is explained in the How show_doc works section below.
+
When generating your docs site, the intermediate output of these pre-processed notebooks and other quarto project files are saved into a directory named _proc/ at the root of your repo. You can inspect the _proc/ directory in order to debug or understand how notebooks are pre-processed.
+
+
+
Quarto
+
Quarto is the mechanism nbdev uses to generate web pages from notebooks. It is useful to visit the Quarto docs and understand how it works. nbdev automatically generates the Quarto configuration files _quarto.yml and sidebar.yml for you.
+
You can override any settings in _quarto.yml by defining a custom.yml file. This is explained further in the Customizing Quarto section. We explain how to customize your sidebar in the Customizing The Sidebar section.
+
+
+
Static Site
+
Quarto has a built-in static site generator that will generate HTML, Javascript and CSS files. These files will be placed in the doc_path directory as specified in your project’s settings.ini file, which is _docs/ by default.
+
+
+
+
Overview
+
Below is a diagram on how these concepts fit together.
+
+
+
+
+
+
flowchart TB
+ %%styles
+ style JN fill:#FFA500
+ style FP fill:#cfe5ed
+ style SF fill:#dff1dd,stroke-dasharray: 5 5;
+ style QMD fill:#7286bb,color:#fff;
+ classDef files fill:#ede8ce ;
+ classDef code fill:#5695c7,color:#fff;
+ classDef container fill:#f9f9f6;
+
+ %% list of nodes
+ FP(<strong>Processing Pipeline</strong>\ntransforms notebook based\non directives and front-matter)
+ E(execnb)
+ SD("show_doc")
+ SS(<strong>Static Site</strong>\nHTML, CSS and Javascript)
+ CF("Intermediate Output is stored in the <code>_procs/</code> directory\n\n<i>(This is a full copy of your Quarto project)</i>")
+ class SD,E code;
+
+ subgraph SF["<strong>Source Files</strong>"]
+ JN([Jupyter\nNotebook])
+ QMD(["Quarto\nMarkdown\n(.qmd)"])
+ end
+
+
+ %% connections to things inside Notebook Processor (NBP)
+ JN -- json --> FP
+ E -. "cell execution" .- SD
+
+ subgraph NBP [" <strong>Notebook Processor\n</strong> "]
+ SD -.- |"render API docs"|FP
+ end
+
+ FP -- modified json with only\nQuarto directives remaining --> CF
+
+ subgraph Quarto [" <strong>Quarto</strong>\n <br>"]
+ direction LR
+ F[[_quarto.yml]] .-> G[[custom.yml]] & H[[sidebar.yml]]
+ class F,G,H files;
+ end
+
+ QMD --"rendered\ndirectly by Quarto\n(no pre-processing required)"--> CF
+ CF --> Quarto
+ Quarto --> SS
+
+ class NBP,CF,Quarto container;
+
+
+
+
+
+
+
+
+
Customizing Quarto
+
You can create a custom.yml file in the same location as your _quarto.yml file to override any values in _quarto.yml. For example, assume your _quarto.yml file looks contains this:
Let’s assume you want to customize your sidebar navigation options such that instead of “floating” for sidebar.style, you wanted your navbar to be “docked”. Additionally, lets assume you want a different background color using the sidebar.background field which is not in the configuration above.
+
To accomplish these tasks, you would create a custom.yml in the same location as _quarto.yml with these contents:
You can also set custom_quarto_yml = True in settings.ini if you wish to edit _quarto.yml directly instead of overriding settings in custom.yml.
+
+
+
+
Customizing The Sidebar
+
By default nbdev automatically generates sidebar.yml, which specifies the tree structure of your sidebar. nbdev infers the tree structure by inspecting the directory structure containing your source files. You can see an example of this by inspecting the folder structure of the notebooks directory in nbdev and the corresponding left-hand sidebar on this website. Leading numbers in filenames and directories are ignored when naming elements of the sidebar (which you can see examples of in this project’s notebooks directory).
+
To customize the sidebar, you must set custom_sidebar = true in settings.ini. This will prevent nbdev from regenerating this file every time the docs are re-built. This way, you an edit this file directly instead of overriding the sidebar with custom.yml.
When your documention website is built, all functions and classes you export to source code will be automatically documented with show_doc. This function outputs a summary of the symbol, showing its signature, docstring, and parameters. For instance, if you have this function:
+
+
def say_gday(
+ greeting:str="G'day", # Greeting to use
+ strine:bool=True, # Use incomprehensible Aussie accent?
+ dropbears:bool=False): # Also warn about drop-bears?
+"Says g'day, the classic Aussie greeting"
+ ...
+
+
This is how it’s rendered in the documentation, by automatically generating a temporary cell containing:
Because this is done automatically any time you build your docs (including automatically by continuous integration), your documentation will always contain current information about your code.
+
You can also document code that’s not created in a notebook, by using show_doc with imported code. For instance, if we wanted to document release_conda, we can import it and call show_doc(release_conda):
+
+
from nbdev.release import release_conda
+show_doc(release_conda)
Create a meta.yaml file ready to be built into a package, and optionally build and upload it
+
+
+
+
+
+
+
+
+
+
+
Type
+
Default
+
Details
+
+
+
+
+
path
+
str
+
conda
+
Path where package will be created
+
+
+
do_build
+
bool_arg
+
True
+
Run conda build step
+
+
+
build_args
+
str
+
+
Additional args (as str) to send to conda build
+
+
+
skip_upload
+
store_true
+
False
+
Skip anaconda upload step
+
+
+
mambabuild
+
store_true
+
False
+
Use mambabuild (requires boa)
+
+
+
upload_user
+
str
+
None
+
Optional user to upload package to
+
+
+
+
+
+
+
Automatic Cell Execution
+
When your documentation is built, all your manually added show_doc cells are run automatically. nbdev also executes all cells containing an import statement, and all exported cells – that way we can be sure that the symbol used in your show_doc cell is available.
+
We don’t, however, execute any other cells. That’s because you wouldn’t want to wait for your entire notebook to run just to build your docs; for instance, your docs might demonstrate training a model which takes hours to complete!
+
This leads to an important rule when authoring nbdev notebooks:
+
+
+
+
+
+
+Warning
+
+
+
+
Do not mix import statements (or calls to show_doc) with other code in a single cell. If you do, all the code in that cell will be executed every time you build your docs, which might lead to errors (since not all previous cells will have been executed.
+
Instead, put your imports in separate cells, and calls to show_doc should contain only that one line of code – the show_doc call.
+
+
+
Note that nbdev automatically hides the actual show_doc(...) line of code. So your users only see the output.
+
+
Forcing Cells To Execute
+
Sometimes you may want to execute additional cells beyond what is automatically executed by nbdev. For instance, on our Getting Started page we show a list of all nbdev commands, automatically generated with nbdev_help. We want this page to always have the most up to date list of commands and docs, so we want it to always execute when the docs are rendered. To do that, add the following directive to the top of a cell:
+
#| exec_doc
+
Alternatively, you can get nbdev to automatically execute all cells when rendering your docs, by adding the following to your notebook frontmatter:
+
---
+exec_all: true
+---
+
+
+
Skipping Execution
+
Likewise, you can instruct nbdev to not execute any cells when rendering your docs with the following front matter:
+
---
+skip_showdoc: true
+---
+
Or ignore execution for a specific cell with this directive:
Many Python developers use sphinx autodoc to automatically document a whole module all at once. Whilst this can work reasonably well, we think there are huge benefits for both developers and users in using nbdev’s approach instead
+
The premise of nbdev’s approach is that your documentation is important enough to be worth you taking the time to think carefully though each thing you want to show your users, what examples you’re going to provide, maybe adding some images to explain more complex ideas, and so forth. Jupyter provides a terrific environment for creating just these kinds of documents. For instance, with Jupyter you can:
+
+
Paste images directly from your clipboard into a cell
+
Insert code and have it executed and the results displayed to users
+
Create a hierarchy of headings to help structure your page
+
…and much more.
+
+
With show_doc, you can insert automatically-updated API details for your library anywhere in a page. That means that you get to decide exactly how your page should look, and what information is provided where. You don’t have to limit yourself to the limits of ASCII art for diagrams, and can include full end-to-end code walk-through of the processes your users will be following.
+
+
+
+
Previewing Your Site Locally
+
You can preview your docs anytime by running nbdev_preview. While in preview mode, you can make updates to notebooks and it will be reflected (after a small delay) in your browser.
+
+
+
Deploying Docs With GitHub Actions
+
If your nbdev project lives in GitHub, we include automation that deploys your documentation site for you on GitHub Pages.
+
nbdev comes bundled with a workflow file that enables this automation. This workflow is automatically triggered whenever you change any of the content in your repo. The following GitHub Actions workflows will run to generate a docs site (in this order):
pages build and deployment: This is an internal built-in workflow that GitHub provides whenever GitHub pages are enabled.
+
+
Should anything go wrong in your page build, you can always look at the logs of these workflows. Like other workflows, these can be found in the Actions tab of your repo:
+
+
+
+
+
+
To read more about GitHub Actions, see their docs.
+
+
+
Deploying Your Docs On Other Platforms
+
You can generate all of the static assets for your site (html, css, etc) by running the command nbdev_docs. After running this command, all of the files for your documentation site will be located in the _docs/ directory (the location is configurable by doc_path in settings.ini) at the root of your repository. This directory is not checked into git and is ignored by .gitignore, but you can use these files to deploy to any hosting platform you want.
+
You can also use the quarto publish command to render your docs on a wide variety of other platforms, which is discussed in the Quarto docs here. After running the command nbdev_docs, the quarto publish command must be run from the root of the _proc/ directory, which is located at the root of your repo. The built-in help of quarto publish provides a good overview of the various targets available:
+
+
+
+
+
+
+Call nbdev_proc_nbs and publish from the _proc/ directory
+
+
+
+
To use quarto publish with nbdev, you must run the nbdev_proc_nbs command to pre-process your notebooks before publishing your docs. As a reminder, nbdev_proc_nbs creates the directory _proc/ at the root of your project that Quarto uses to render your site.
+
For example, to publish a site to Netlify you can run the following command from the root of your repo:
+
nbdev_proc_nbs&&cd _proc &&quarto publish netlify
+
+
+
+
!quarto publish -h
+
+
+ Usage: quarto publish [provider] [path]
+ Version: 1.1.75
+
+
+ Description:
+
+ Publish a document or project. Available providers include:
+
+ - Quarto Pub (quarto-pub)
+ - GitHub Pages (gh-pages)
+ - RStudio Connect (connect)
+
+ - Netlify (netlify)
+ Accounts are configured interactively during publishing.
+ Manage/remove accounts with: quarto publish accounts
+
+ Options:
+
+ -h, --help - Show this help.
+ --id <id> - Identifier of content to publish
+ --server <server> - Server to publish to
+ --token <token> - Access token for publising provider
+ --no-render - Do not render before publishing.
+ --no-prompt - Do not prompt to confirm publishing destination
+ --no-browser - Do not open a browser to the site after publishing
+ --log <level> - Path to log file
+ --log-level <level> - Log level (info, warning, error, critical)
+ --log-format <format> - Log format (plain, json-stream)
+ --quiet - Suppress console output.
+
+ Commands:
+
+ help [command] - Show this help or the help of a sub-command.
+
+ Examples:
+
+ Publish project (prompt for provider): quarto publish
+ Publish document (prompt for provider): quarto publish document.qmd
+ Publish project to Netlify: quarto publish netlify
+ Publish with explicit target: quarto publish netlify --id DA36416-F950-4647-815C-01A24233E294
+ Publish project to GitHub Pages: quarto publish gh-pages
+ Publish project to RStudio Connect: quarto publish connect
+ Publish with explicit credentials: quarto publish connect --server example.com --token 01A24233E294
+ Publish without confirmation prompt: quarto publish --no-prompt
+ Publish without rendering: quarto publish --no-render
+ Publish without opening browser: quarto publish --no-browser
+ Manage/remove publishing accounts: quarto publish accounts
+
These explanations provide background information on how nbdev is designed and how it works. They are designed to help people who want to more deeply understand the nbdev system.
Popular Python documentation standards such as numpy docstrings and sphinx facilitate documentation of source code with docstrings, which are limited in their expressiveness and functionality. Notebooks, on the other hand, offer a richer medium for authoring documentation (with markdown and code cells) compared to docstrings, and unlock new ways of documenting your code that are otherwhise infeasible.
+
Furthermore, traditional testing frameworks such as pytest and unittest encourage writing tests in a separate context that is disjointed from the associated source code and documentation. With nbdev, you write tests in the same context as your source code and documentation, such that tests can optionally become part of the narrative within your documentation.
+
Since nbdev allows your colleagues and users to view the tests, code and documentation in a single context with a REPL that invites experimentation, the way you author code, documentation and tests in nbdev are very different than traditional software development workflows.
+
In Notebook Best Practices we compare a function coded, tested, and documented in nbdev versus ordinary .py files. Here are a few of the challenges we faced with other approaches that led us to using nbdev. In .py files:
+
+
Docstrings repeat information that is already contained in the function signature, such as names of parameters, default values, and types
+
Examples are manually entered. This requires the author to copy and paste both the code and outputs. Furthermore, the author must manually re-enter or change these examples anytime the code changes, which is an error-prone process
+
Examples are limited to short code snippets and cannot contain rich output like plots or graphics
+
Examples cannot have prose interleaved with code except for code comments
+
Readers of your code must copy and paste contents of the docstring if they wish to reproduce the examples.
+ Create delightful software with Jupyter Notebooks
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
nbdev is a notebook-driven development platform. Simply write notebooks with lightweight markup and get high-quality documentation, tests, continuous integration, and packaging for free!
+
nbdev makes debugging and refactoring your code much easier than in traditional programming environments since you always have live objects at your fingertips. nbdev also promotes software engineering best practices because tests and documentation are first class.
+
+
Documentation is automatically generated using Quarto and hosted on GitHub Pages. Docs support LaTeX, are searchable, and are automatically hyperlinked (including out-of-the-box support for many packages via nbdev-index)
+
Publish packages to PyPI and conda as well as tools to simplify package releases. Python best practices are automatically followed, for example, only exported objects are included in __all__
+
Two-way sync between notebooks and plaintext source code allowing you to use your IDE for code navigation or quick edits
+
Tests written as ordinary notebook cells are run in parallel with a single command
+
Continuous integration out-of-the-box with GitHub Actions that run your tests and rebuild your docs
+
Git-friendly notebooks with Jupyter/Git hooks that clean unwanted metadata and render merge conflicts in a human-readable format
+
… and much more!
+
+
+
Install
+
nbdev works on macOS, Linux, and most Unix-style operating systems. It works on Windows under WSL, but not under cmd or Powershell.
+
You can install nbdev with pip:
+
pip install nbdev
+
… or with conda (or mamba):
+
conda install -c fastai nbdev
+
Note that nbdev must be installed into the same Python environment that you use for both Jupyter and your project.
+
+
+
How to use nbdev
+
The best way to learn how to use nbdev is to complete either the written walkthrough or video walkthrough:
+
+
Alternatively, there’s a shortened version of the video walkthrough with coding sections sped up using the unsilence Python library – it’s 27 minutes faster, but a bit harder to follow.
+
You can also run nbdev_help from the terminal to see the full list of available commands:
+
+
!nbdev_help
+
+
nbdev_bump_version Increment version in settings.ini by one
+nbdev_changelog Create a CHANGELOG.md file from closed and labeled GitHub issues
+nbdev_clean Clean all notebooks in `fname` to avoid merge conflicts
+nbdev_conda Create a `meta.yaml` file ready to be built into a package, and optionally build and upload it
+nbdev_create_config Create a config file.
+nbdev_docs Create Quarto docs and README.md
+nbdev_export Export notebooks in `path` to Python modules
+nbdev_filter A notebook filter for Quarto
+nbdev_fix Create working notebook from conflicted notebook `nbname`
+nbdev_help Show help for all console scripts
+nbdev_install Install Quarto and the current library
+nbdev_install_hooks Install Jupyter and git hooks to automatically clean, trust, and fix merge conflicts in notebooks
+nbdev_install_quarto Install latest Quarto on macOS or Linux, prints instructions for Windows
+nbdev_merge Git merge driver for notebooks
+nbdev_migrate Convert all markdown and notebook files in `path` from v1 to v2
+nbdev_new Create an nbdev project.
+nbdev_prepare Export, test, and clean notebooks, and render README if needed
+nbdev_preview Preview docs locally
+nbdev_proc_nbs Process notebooks in `path` for docs rendering
+nbdev_pypi Create and upload Python package to PyPI
+nbdev_readme Create README.md from readme_nb (index.ipynb by default)
+nbdev_release_both Release both conda and PyPI packages
+nbdev_release_gh Calls `nbdev_changelog`, lets you edit the result, then pushes to git and calls `nbdev_release_git`
+nbdev_release_git Tag and create a release in GitHub for the current version
+nbdev_requirements Writes a `requirements.txt` file to `directory` based on settings.ini.
+nbdev_sidebar Create sidebar.yml
+nbdev_test Test in parallel notebooks matching `path`, passing along `flags`
+nbdev_trust Trust notebooks matching `fname`
+nbdev_update Propagate change in modules matching `fname` to notebooks that created them
+nbdev_update_license Allows you to update the license of your project.
+
+
+
+
+
FAQ
+
+
Q: What is the warning “Found a cell containing mix of imports and computations. Please use separate cells”?
+
A: You should not have cells that are not exported, and contain a mix of import statements along with other code. For instance, don’t do this in a single cell:
+
import some_module
+some_module.something()
+
Instead, split this into two cells, one which does import some_module, and the other which does some_module.something().
+
The reason for this is that when we create your documentation website, we ensure that all of the signatures for functions you document are up to date, by running the imports, exported cells, and show_doc functions in your notebooks. When you mix imports with other code, that other code will be run too, which can cause errors (or at least slowdowns) when creating your website.
+
+
+
Q: Why is nbdev asking for root access? How do I install Quarto without root access?
+
A: When you setup your first project, nbdev will attempt to automatically download and install Quarto for you. This is the program that we use to create your documentation website.
+
Quarto’s standard installation process requires root access, and nbdev will therefore ask for your root password during installation. For most people, this will work fine and everything will be handled automatically – if so, you can skip over the rest of this section, which talks about installing without root access.
+
If you need to install Quarto without root access on Linux, first cd to wherever you want to store it, then download Quarto, and type:
To use this non-root version of Quarto, you’ll need ~/.local/bin in your PATH environment variable. (Alternatively, change the ln -s step to place the symlink somewhere else in your path.)
+
+
+
Q: Someone told me not to use notebooks for “serious” software development!
If you want to contribute to nbdev, be sure to review the contributions guidelines. This project adheres to fastai’s code of conduct. By participating, you are expected to uphold this code. In general, we strive to abide by generally accepted best practices in open-source software development.
+
Make sure you have nbdev’s git hooks installed by running nbdev_install_hooks in the cloned repository.
Traditional programming environments throw away the result of your exploration in REPLs or notebooks. nbdev makes exploration an integral part of your workflow, all while promoting software engineering best practices.
+
+
+
+
+
Beautiful technical documentation and scientific articles with Quarto
+
+
+
+
Out-of-the-box continuous integration with GitHub Actions
+
+
+
+
Publish code to PyPI and conda, and prose to GitHub Pages
+
+
+
+
Two-way sync with your favourite IDEs
+
+
+
+
Write prose, code, and tests in notebooks — no context-switching
+
+
+
+
Git-friendly notebooks: human-readable merge conflicts; no unwanted metadata
+
+
+
+
+
+
Here’s what experts are saying
+
+
+
+
+
Chris Lattner
+
Inventor of Swift and LLVM
+
I really do think [nbdev] is a huge step forward for programming environments.
+
+
+
+
Fernando Pérez
+
Creator of Jupyter
+
[nbdev] should be celebrated and used a lot more — I have kept a tab with your original nbdev blog post open for months in Chrome because of how often I refer to it and point others to this work.
+
+
+
+
David Berg
+
Software Engineer, Netflix
+
Prior to using nbdev, documentation was the most cumbersome aspect of our software development process… Using nbdev allows us to spend more time creating rich prose around the many code snippets guaranteeing the whole experience is robust.
nbdev has turned what was once a chore into a natural extension of the notebook-based testing we were already doing.
+
+
+
+
Erik Gaasedelen
+
Software Engineer, Lyft
+
I use this in production at my company. It’s an awesome tool… nbdev streamlines everything so I can write docs, tests, and code all in one place… The packaging is also really well thought out.
From my point of view it is close to a Pareto improvement over traditional Python library development.
+
+
+
+
Roxanna Pourzand
+
Product Manager, Transform
+
We’re so excited about using nbdev. Our product is technical so our resulting documentation includes a lot of code-based examples. Before nbdev, we had no way of maintaining our code examples and ensuring that it was up-to-date for both command inputs and outputs. It was all manual. With nbdev, we now have this under control in a sustainable way. Since we’ve deployed these docs, we also had a situation where we were able to identify a bug in one of our interfaces, which we found by seeing the error that was output in the documentation.
+
+
+
+
Hugo Bowne-Anderson
+
Head of Developer Relations, Outerbounds
+
Nbdev has transformed the way we write documentation. Gone are the days of worrying about broken code examples when our API changes or due to human errors associated with copying & pasting code into markdown files. The authoring experience of nbdev is also powerful, allowing us to write prose and live code in a unified interface, which allows more experimentation with technical content. On top of this, nbdev allows us to include unit tests in our documentation which mitigates the burden of maintaining the docs over time.
