Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

An exception occurred when the kernel restarted, causing it to keep restarting #734

Open
icankeep opened this issue Jan 10, 2022 · 3 comments

Comments

@icankeep
Copy link

image

kernel is restarting

image

[E 2022-01-10 17:45:38.522 MtSingleUserLabApp handlers:82] Exception restarting kernel
    Traceback (most recent call last):
      File "/conda/envs/notebook/lib/python3.6/site-packages/jupyter_server/services/kernels/handlers.py", line 80, in post
        await km.restart_kernel(kernel_id)
      File "/conda/envs/notebook/lib/python3.6/site-packages/mtjupyter_singleuser/kernelmanager.py", line 92, in restart_kernel
        await super(AsyncMappingKernelManager, self).restart_kernel(kernel_id, now)
      File "/conda/envs/notebook/lib/python3.6/site-packages/jupyter_server/services/kernels/kernelmanager.py", line 384, in restart_kernel
        await ensure_async(self.pinned_superclass.restart_kernel(self, kernel_id, now=now))
      File "/conda/envs/notebook/lib/python3.6/site-packages/jupyter_server/utils.py", line 189, in ensure_async
        result = await obj
      File "/conda/envs/notebook/lib/python3.6/site-packages/jupyter_client/manager.py", line 478, in _async_restart_kernel
        await ensure_async(self.shutdown_kernel(now=now, restart=True))
      File "/conda/envs/notebook/lib/python3.6/site-packages/jupyter_client/utils.py", line 34, in ensure_async
        return await obj
      File "/conda/envs/notebook/lib/python3.6/site-packages/jupyter_client/manager.py", line 434, in _async_shutdown_kernel
        await ensure_async(self.interrupt_kernel())
      File "/conda/envs/notebook/lib/python3.6/site-packages/jupyter_client/utils.py", line 34, in ensure_async
        return await obj
      File "/conda/envs/notebook/lib/python3.6/site-packages/jupyter_client/manager.py", line 542, in _async_interrupt_kernel
        raise RuntimeError("Cannot interrupt kernel. No kernel is running!")
    RuntimeError: Cannot interrupt kernel. No kernel is running!
@icankeep
Copy link
Author

I don't know how to reproduce this situation.

I think if the kernel is not found, it should not throw an exception, but just print the log, otherwise it will affect the startup process

@kevin-bates
Copy link
Member

I don't know how to reproduce this situation.

Might this be the result of a subsequent restart request, while the first was in the process of completing?

I think if the kernel is not found, it should not throw an exception, but just print the log, otherwise it will affect the startup process

This seems reasonable, although I'd be inclined to check self.has_kernel in the shutdown method, prior to its sequence of interrupt, request-shutdown, etc. Perhaps in such cases (when shutdown_kernel encounters a false has_kernel value), we simply log a warning and return, since there really isn't much else we can do (although I haven't looked closely relative to that last portion).

@Zsailer is also applying "pending-state" support to the shutdown operation so he may have some insights here as well.

@Zsailer
Copy link
Member

Zsailer commented Jan 12, 2022

I've seen this error with slow starting kernels. If you try to restart a kernel that takes a long time to start (this is often the case with remote kernels), you'll get this error.

I agree, we can likely log this issue without raising an exception. In a "pending kernels" world, proposed by #732, you won't run into this issue. Restarting a slow starting kernel will be blocked.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants