[BUG] - Flaky local integration tests failing with paramiko.ssh_exception.AuthenticationException

[marcelovilla]

Describe the bug

I've noticed that sporadically, our local integration tests might fail with the following error:

------------------------------ Captured log setup ------------------------------
2024-10-17 06:18:35      INFO 1944 transport: Connected (version 2.0, client AsyncSSH_2.13.2)
2024-10-17 06:18:35      INFO 1944 transport: Auth banner: b'Starting your server...'
2024-10-17 06:18:36      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:36      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:37      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:37      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:38      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:38      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:39      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:39      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:40      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:40      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:41      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:41      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:42      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:42      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:43      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:43      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:44      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:44      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:45      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:46      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:46      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:47      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:47      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:48      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:48      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:49      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:49      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:50      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:50      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:51      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:51      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:52      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:52      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:53      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:53      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:54      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:54      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:55      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:55      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:56      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:56      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:57      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:57      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:58      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:58      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:18:59      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:00      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:00      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:01      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:01      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:02      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:02      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:03      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:03      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:04      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:04      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:05      INFO 1944 transport: Auth banner: b'.'
2024-10-17 06:19:05      INFO 1944 transport: Auth banner: b'failed to start server on time!\n'
2024-10-17 06:19:05      INFO 1944 transport: Authentication (password) failed.
=========================== short test summary info ============================
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_simple_jupyterhub_ssh - paramiko.ssh_exception.AuthenticationException: Authentication failed.
==================== 27 passed, 1 error in 73.40s (0:01:13) ====================

Usually a simple re-trigger of the GHA workflow will work. See for example:

• https://github.com/nebari-dev/nebari/actions/runs/11379267190/attempts/1?pr=2773 (failed)
• https://github.com/nebari-dev/nebari/actions/runs/11379267190/attempts/2?pr=2773 (succeeded after re-trigger)

Expected behavior

Local integration tests should not fail because of an authentication error, specially if suggested changes in a PR don't involve that component.

OS and architecture in which you are running Nebari

GHA

How to Reproduce the problem?

This issue happens only sporadically and it's not consistently reproducible.

Command output

No response

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response

[viniciusdc]

Usually, the pod is not ready in time, and paramiko starts the connection attempts while the container begins, leading to a timeout. Because the connection access triggers the pod, I suggest adding a backoff retry workflow instead of just increasing the timeout.

the paramiko client might already have parameter of such.