Investigate How to Handle Layered Extensions #3

blink1073 · 2020-07-07T19:10:02Z

Look into the logic for nbextensions and come up with a game plan for the new labextensions

Specifically,

What is the discovery mechanism?
What is the command line interface?
What does the REST API look like (notionally /lab/ext/<foo>)?

Create as a server extension here and then port to jupyterlab.

The text was updated successfully, but these errors were encountered:

blink1073 · 2020-07-07T19:55:13Z

nbextensions are documented here

blink1073 · 2020-07-07T19:55:54Z

blink1073 · 2020-07-07T19:56:39Z

Example:

$ jupyter nbextension list
Known nbextensions:
  config dir: /Users/stslve/miniconda/envs/jlab-pip-2.0/etc/jupyter/nbconfig
    notebook section
      nbdime/index  enabled
      - Validating: OK

blink1073 · 2020-07-07T19:58:19Z

$ jupyter nbextension help
Work with Jupyter notebook extensions

Subcommands
-----------

Subcommands are launched as `jupyter nbextension cmd [args]`. For information on
using subcommand 'cmd', do: `jupyter nbextension cmd -h`.

install
    Install an nbextension
enable
    Enable an nbextension
disable
    Disable an nbextension
uninstall
    Uninstall an nbextension
list
    List nbextensions

Options
-------

Arguments that take values are actually convenience aliases to full
Configurables, whose aliases are listed on the help line. For more information
on full configurables, see '--help-all'.

--debug
    set log level to logging.DEBUG (maximize logging output)
--user
    Apply the operation only for the given user
--system
    Apply the operation system-wide
--sys-prefix
    Use sys.prefix as the prefix for installing nbextensions (for environments, packaging)
--py
    Install from a Python package
--python
    Install from a Python package
--log-level=<Enum> (Application.log_level)
    Default: 30
    Choices: (0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL')
    Set the log level by value or name.
--config=<Unicode> (JupyterApp.config_file)
    Default: ''
    Full path of a config file.

To see all available configurables, use `--help-all`

Examples
--------

    jupyter nbextension list                          # list all configured nbextensions
    jupyter nbextension install --py <packagename>    # install an nbextension from a Python package
    jupyter nbextension enable --py <packagename>     # enable all nbextensions in a Python package
    jupyter nbextension disable --py <packagename>    # disable all nbextensions in a Python package
    jupyter nbextension uninstall --py <packagename>  # uninstall an nbextension in a Python package

$ jupyter nbextension install help
Install Jupyter notebook extensions

Usage

    jupyter nbextension install path|url [--user|--sys-prefix]

This copies a file or a folder into the Jupyter nbextensions directory. If a URL
is given, it will be downloaded. If an archive is given, it will be extracted
into nbextensions. If the requested files are already up to date, no action is
taken unless --overwrite is specified.

Options
-------

Arguments that take values are actually convenience aliases to full
Configurables, whose aliases are listed on the help line. For more information
on full configurables, see '--help-all'.

--debug
    set log level to logging.DEBUG (maximize logging output)
--user
    Apply the operation only for the given user
--system
    Apply the operation system-wide
--sys-prefix
    Use sys.prefix as the prefix for installing nbextensions (for environments, packaging)
--py
    Install from a Python package
--python
    Install from a Python package
--overwrite
    Force overwrite of existing files
--symlink
    Create symlink instead of copying files
-s
    Create symlink instead of copying files
--log-level=<Enum> (Application.log_level)
    Default: 30
    Choices: (0, 10, 20, 30, 40, 50, 'DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL')
    Set the log level by value or name.
--config=<Unicode> (JupyterApp.config_file)
    Default: ''
    Full path of a config file.
--prefix=<Unicode> (InstallNBExtensionApp.prefix)
    Default: ''
    Installation prefix
--nbextensions=<Unicode> (InstallNBExtensionApp.nbextensions_dir)
    Default: ''
    Full path to nbextensions dir (probably use prefix or user)
--destination=<Unicode> (InstallNBExtensionApp.destination)
    Default: ''
    Destination for the copy or symlink

To see all available configurables, use `--help-all`

Examples
--------

    jupyter nbextension install /path/to/myextension

blink1073 · 2020-07-07T20:10:50Z

In classic notebook, extensions are JS modules that end up in share/jupyter/nbextensions and are configured in etc/jupyter/nbconfig/notebook.d/. They can be manually installed from a url, a symlink, or a python package. I think we should stick with only Python packages that are enabled, similar to how server extensions work.

blink1073 · 2020-07-07T20:13:45Z

The proposed layout is share/jupyter/lab/modules/<foo> for the module bundles and etc/jupyter/jupyter_lab_config.d/<foo>.json for config.

blink1073 · 2020-07-07T20:20:37Z

Hmm, does install even make sense then, jupyter serverextension only has list, enable, and disable

blink1073 · 2020-07-07T20:27:21Z

Dev workflow assuming no labextension install:

Extension author runs pip install -e . for their extension foo created from our cookiecutter, that uses jupyter_packaging. This builds the extension if needed, copies data files to share/jupyter/lab/modules/foo and creates config file etc/share/jupyter/jupyter_lab_config.d/foo.json
Extension has metadata about which folder has their static files
Extension author runs jupyter lab watch
Extension author makes changes to source files, which are copied to share/jupyter/lab/modules/foo
Extension author refreshes the page to pick up changes from their module

blink1073 · 2020-07-07T21:24:48Z

Here's how it is used in the notebook server: https://github.com/jupyter/notebook/blob/0df10dee3ee7e8c0f56f879b69d37070fccca1c7/notebook/notebookapp.py#L362

blink1073 · 2020-07-07T21:43:01Z

Proposed layout:

etc/jupyter/labextensions.d/
share/jupyter/labextensions/foo/schemas
                                                    /themes
                                                   /static
share/jupyter/lab/settings/page_config.json
                                          /overrides.json

TODO for building a compiled version
<fill in static, staging, schema, etc. (maybe settings/build_config.json)

TODO later, make this layer-able
~/.jupyter/lab/user-settings
~/.jupyter/lab/workspaces

jasongrout · 2020-07-07T23:00:49Z

Notes from our conversation today:

How will directories work?

JUPYTERLAB_CORE_DIR
default is /conda/env/share/jupyter/lab/static
core_mode is /site_packages/jupyterlab/static
sys admin sets to /opt/mylab/jupyter/static

application directory is prefix/share/jupyter/lab

The question is: do we load dynamic extensions on top of this?

--core-mode: loads site-package distributed bundle
--no-dynamic-extensions: do not load any dynamic extensions

Current layer stays the same

Except now we build module federation capable bundles, and labextension list lists the dynamic modules

jupyter labextension install: installs an npm package as a package in the application dir bundle

jupyter labextension list: lists all extensions, including dynamic extensions, and indicates when one overrides another.

jupyter lab build: builds the application dir bundle

jupyter lab clean: cleans the application dir bundle

These directories also stay the same:

share/jupyter/lab/settings/page_config.json
                           /overrides.json

Dynamic packages

Distributed in any way, i.e., conda package, pip package, debian package, etc.

Webpack module federation bundle is put into prefix/share/jupyter/labextensions/<MODULE>/

Inside this directory, we have share/jupyter/labextensions/<MODULE>

/schemas -settings schemas
/themes - css
/static - the module federation bundle, including a remoteEntrydle, including a remoteEntry.js

Follow the Jupyter path layering, where more plugins in more specific layers override more general layers.

Enabling and disabling extensions works exactly like in server extensions, i.e., conf.d files in etc/jupyter/labextensions.d/. In general, dynamic extensions override the base application bundled extensions.

index.js

  if ext_disabled:
      do nothing
  elif dynamic_version_available:
      pull in remote
  else:
      pull in local

Templated into page config:

List of dynamic extensions installed (server will proxy the URLs to the right directory)
enabled/disabled extension list
the current stuff as well

Supporting extensions depending on other extensions:

Dynamic extensions get this automatically by depending on the relevant other python packages
We could implement extension dependency tracing for npm extensions, i.e., see if any of your dependencies are extensions, load them, recurse.

blink1073 · 2020-07-15T22:24:36Z

The rest endpoint should be /labextensions/<foo> to mirror nextensions

blink1073 mentioned this issue Jul 7, 2020

Expand the settings handler to account for extensions #4

Closed

blink1073 mentioned this issue Jul 22, 2020

Add labextensions handler #25

Merged

blink1073 added the reference label Aug 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Investigate How to Handle Layered Extensions #3

Investigate How to Handle Layered Extensions #3

blink1073 commented Jul 7, 2020 •

edited

Loading

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020 •

edited

Loading

blink1073 commented Jul 7, 2020 •

edited

Loading

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020 •

edited

Loading

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020

jasongrout commented Jul 7, 2020

blink1073 commented Jul 15, 2020

Investigate How to Handle Layered Extensions #3

Investigate How to Handle Layered Extensions #3

Comments

blink1073 commented Jul 7, 2020 • edited Loading

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020 • edited Loading

blink1073 commented Jul 7, 2020 • edited Loading

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020 • edited Loading

blink1073 commented Jul 7, 2020

blink1073 commented Jul 7, 2020

jasongrout commented Jul 7, 2020

How will directories work?

Current layer stays the same

Dynamic packages

Templated into page config:

Supporting extensions depending on other extensions:

blink1073 commented Jul 15, 2020

blink1073 commented Jul 7, 2020 •

edited

Loading

blink1073 commented Jul 7, 2020 •

edited

Loading

blink1073 commented Jul 7, 2020 •

edited

Loading

blink1073 commented Jul 7, 2020 •

edited

Loading