CI is Flakey

[blink1073]

Description

The "Test Full Install" jobs frequently time out, seemingly because of an interaction between virtual envs, asyncio and pytest during teardown. We could try making the full install test a separate job that uses an sdist produced by a previous job to better approximate the standard tests.

[Zsailer]

Flakey-ness seems to be happening other places as well, e.g. https://github.com/jupyter-server/jupyter_server/runs/5038246226?check_suite_focus=true.

It definitely happens in the "Test Full Install" workflow more often than others, but it doesn't appear to be isolated there.

[blink1073]

For the builds that are timing out, we can use pytest-timeout the thread method to get a traceback on CI.

[blink1073]

After #689, we're seeing failures like the following:

   File "/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 116, in __del__
    self.context.destroy()
  File "/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/zmq/sugar/context.py", line 243, in destroy
    self.term()
  File "/opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/zmq/sugar/context.py", line 190, in term
    return super(Context, self).term()

+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
tests/services/kernels/test_api.py::test_kernel_handler[jp_server_config2]

It is also failing:

 tests/test_terminal.py::test_terminal_create_with_cwd <- ../../../../../opt/hostedtoolcache/Python/3.9.10/x64/lib/python3.9/site-packages/jupyter_server/tests/test_terminal.py FAILED

[blink1073]

In today's dev meeting, we thought this might have something to do with changes in jupyter/jupyter_client#732.

It could also be related to ipython/ipykernel#853

[echarles]

We hit this right now on #165. I consider this issue is not relevant that PR. Would be great to fix this though.

[minrk]

Since one of the flaky tests I saw related to a hanging Context.term, that always means a zmq socket was not properly closed, so changes to kernel management are very likely to be relevant.

[blink1073]

These tests are no longer flaky, due to some cleanup work here and in Jupyter Client.