Async cell execution #541

davidbrochart · 2020-02-11T16:26:23Z

This uses the asynchronous cell execution mode of nbclient, which requires an async client from jupyter_client. Also, nbconvert has to enable async in Jinja.

davidbrochart · 2020-02-12T10:08:13Z

This can't work with Python 3.5 because of Jinja async support (see pallets/jinja#1058).

maartenbreddels · 2020-03-02T15:41:45Z

Running $ time py.test tests/app/ before and after this PR

py.test tests/app/  21.50s user 5.80s system 51% cpu 53.168 total
py.test tests/app/  16.38s user 4.25s system 87% cpu 23.649 total

So less time, and more cpu usage 🚀
(I did rebase on master, since it has an extra test)

maartenbreddels · 2020-03-02T16:01:04Z

I cannot seem to break it yet, if you rebase on master, you'll also get the tests/app/many_iopub_messages_test.py test, which is an important one to pass, since it tests one of the issues of #534, but it passes locally, great work!

davidbrochart · 2020-03-10T10:17:06Z

Trigger CI.

jf--- · 2020-03-12T18:46:58Z

can't wait for this to land -- both significant performance gains at the "cost" of considerably cleaner code --- duly impressive

davidbrochart · 2020-04-23T14:19:08Z

Trigger CI.

davidbrochart · 2020-04-23T14:34:03Z

Trigger CI.

davidbrochart · 2020-04-23T15:48:15Z

The Jupyter server app has to use an AsyncMappingKernelManager. If that is acceptable, this PR is ready to be merged.

… AsyncKernelClient

maartenbreddels

Awesome work, really glad to see a lot of code removed. I wonder if we want or should try to go for nbconvert5 compatibility or not, if possible.

Mosty small comments.

maartenbreddels · 2020-05-04T11:19:23Z

share/jupyter/voila/templates/default/nbconvert_templates/lab.tpl

@@ -1,4 +1,4 @@
-{%- extends 'display_priority.tpl' -%}


With jupyter/nbconvert#1173 we could again use the tpl, if we care about backwards compatibility

I reverted to .tpl.

setup.py

voila/handler.py

maartenbreddels · 2020-05-04T11:36:44Z

voila/handler.py

+        km.client_class = 'jupyter_client.asynchronous.AsyncKernelClient'
+        self.executor = VoilaExecutor(nb, km=km, config=self.traitlet_config)
+        self.executor.kc = km.client()
+        self.executor.kc.start_channels()


I am/was a bit worried about this. I remember when I just started adding an async client that the jupyter_server part used the same client. Maybe that was indeed because I configured it to be async.
Are you saying that both jupyter_server and voila have their own client to the kernel, jupyter_server does whatever it does, and we do it async?
Did you check if https://github.com/voila-dashboards/voila/blob/master/notebooks/basics.ipynb works with both the voila app and server extension?

Yes it works both with the voila app and the server extension. I'm not sure I understand what you mean, Voila has its own kernel anyway, right?

Well, the kernel is also known to jupyter server, which is responsible for the WebSocket connection. So jupyter server had a connection to the kernel as well (actually it may be in parallel to us)

could we use nbclient's async context manager async with self.async_setup_kernel() instead of manually doing

self.executor.kc = km.client() self.executor.kc.start_channels()

.github/workflows/build.yml

maartenbreddels

I think the whole PR is 'atomic' enough to be 1 commit, shall we squash and merge?

.github/workflows/main.yml

setup.py

voila/execute.py

SylvainCorlay · 2020-05-05T13:52:26Z

voila/exporter.py


    @property
    def environment(self):
+        self.enable_async = True


Maybe add a comment to say what this is?

SylvainCorlay · 2020-05-06T08:18:14Z

Weird, there is a test failure over a new line character on Python 3.8 - and also I cannot restart the build.

davidbrochart · 2020-05-06T08:22:53Z

Trigger CI.

SylvainCorlay · 2020-05-06T08:23:52Z

Why Can't I restart CI from the travis UI?

davidbrochart · 2020-05-06T08:24:17Z

No idea.

SylvainCorlay · 2020-05-06T09:06:38Z

🎉

This was referenced Feb 28, 2020

Release jupyter/nbconvert#1206

Closed

Release jupyter/nbclient#31

Closed

davidbrochart force-pushed the nbclient branch 3 times, most recently from 499e6c6 to 86916d8 Compare March 3, 2020 10:15

davidbrochart closed this Mar 10, 2020

davidbrochart reopened this Mar 10, 2020

davidbrochart added 2 commits March 26, 2020 07:17

Async cell execution

1e36145

Use my branch of nbclient

5effb39

davidbrochart force-pushed the nbclient branch 7 times, most recently from 5514bab to 52ce077 Compare March 26, 2020 09:31

Pin jupyter_client>=6.1.1

fd54cb7

davidbrochart force-pushed the nbclient branch from 52ce077 to fd54cb7 Compare March 26, 2020 09:43

Pin nbclient>=0.2.0

d4c55af

davidbrochart mentioned this pull request Apr 8, 2020

nbconvert<6 ? jupyter-server/jupyter_server#198

Closed

davidbrochart added 5 commits April 10, 2020 10:20

Use AsyncMappingKernelManager

b51fc7e

Install pytest-tornasync

d891e53

Pin jupyter_client>=6.1.3

b5c2805

Pin jupyter_server>=0.3.0

50f467d

Add libgbm to fix puppeteer issue

8d4eba6

davidbrochart reopened this Apr 23, 2020

davidbrochart closed this Apr 23, 2020

davidbrochart reopened this Apr 23, 2020

davidbrochart added 2 commits April 23, 2020 16:42

Update actions/checkout

1cb71e5

Launch server app with AsyncMappingKernelManager

a2c1dea

davidbrochart changed the title ~~[WIP] Async cell execution~~ Async cell execution Apr 23, 2020

Support AsyncMappingKernelManager and MappingKernelManager by forcing…

0b32611

… AsyncKernelClient

maartenbreddels reviewed May 4, 2020

View reviewed changes

maartenbreddels mentioned this pull request May 4, 2020

Fix timeout=None #589

Closed

Changes according to review

a6a793f

maartenbreddels approved these changes May 4, 2020

View reviewed changes

SylvainCorlay reviewed May 5, 2020

View reviewed changes

.github/workflows/main.yml Show resolved Hide resolved

SylvainCorlay reviewed May 5, 2020

View reviewed changes

setup.py Show resolved Hide resolved

SylvainCorlay reviewed May 5, 2020

View reviewed changes

voila/execute.py Outdated Show resolved Hide resolved

Pass store_history to async_execute_cell()

3506b68

SylvainCorlay reviewed May 5, 2020

View reviewed changes

voila/exporter.py

@property

def environment(self):

self.enable_async = True

Copy link

Member

SylvainCorlay May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a comment to say what this is?

Add comment for jinja async template execution

6b55fe6

davidbrochart closed this May 6, 2020

davidbrochart reopened this May 6, 2020

SylvainCorlay merged commit 1c9ca89 into voila-dashboards:master May 6, 2020

davidbrochart deleted the nbclient branch May 14, 2020 08:38

maartenbreddels mentioned this pull request Aug 31, 2020

Performance: nbconvert blocks tornado ioloop #363

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Async cell execution #541

Async cell execution #541

davidbrochart commented Feb 11, 2020

davidbrochart commented Feb 12, 2020

maartenbreddels commented Mar 2, 2020

maartenbreddels commented Mar 2, 2020

davidbrochart commented Mar 10, 2020

jf--- commented Mar 12, 2020

davidbrochart commented Apr 23, 2020

davidbrochart commented Apr 23, 2020

davidbrochart commented Apr 23, 2020 •

edited

Loading

maartenbreddels left a comment

maartenbreddels May 4, 2020

davidbrochart May 4, 2020

maartenbreddels May 4, 2020

davidbrochart May 4, 2020

maartenbreddels May 4, 2020

SylvainCorlay May 5, 2020

maartenbreddels left a comment

SylvainCorlay May 5, 2020

SylvainCorlay commented May 6, 2020

davidbrochart commented May 6, 2020

SylvainCorlay commented May 6, 2020

davidbrochart commented May 6, 2020

SylvainCorlay commented May 6, 2020

Async cell execution #541

Async cell execution #541

Conversation

davidbrochart commented Feb 11, 2020

davidbrochart commented Feb 12, 2020

maartenbreddels commented Mar 2, 2020

maartenbreddels commented Mar 2, 2020

davidbrochart commented Mar 10, 2020

jf--- commented Mar 12, 2020

davidbrochart commented Apr 23, 2020

davidbrochart commented Apr 23, 2020

davidbrochart commented Apr 23, 2020 • edited Loading

maartenbreddels left a comment

Choose a reason for hiding this comment

maartenbreddels May 4, 2020

Choose a reason for hiding this comment

davidbrochart May 4, 2020

Choose a reason for hiding this comment

maartenbreddels May 4, 2020

Choose a reason for hiding this comment

davidbrochart May 4, 2020

Choose a reason for hiding this comment

maartenbreddels May 4, 2020

Choose a reason for hiding this comment

SylvainCorlay May 5, 2020

Choose a reason for hiding this comment

maartenbreddels left a comment

Choose a reason for hiding this comment

SylvainCorlay May 5, 2020

Choose a reason for hiding this comment

SylvainCorlay commented May 6, 2020

davidbrochart commented May 6, 2020

SylvainCorlay commented May 6, 2020

davidbrochart commented May 6, 2020

SylvainCorlay commented May 6, 2020

davidbrochart commented Apr 23, 2020 •

edited

Loading