WebSocketBadStatusException: Handshake status 500 Internal Server Error: container not found

[craigwalton-dsit]

Migrated from internal repo.
Complete stack trace and logs (sensitive) https://github.com/AI-Safety-Institute/aisi-inspect-tools/issues/154
Original date: 27 Oct 2024

WebSocketBadStatusException: Handshake status 500 Internal Server Error -+-+- {'content-length': '31', 'content-type': 'text/plain; charset=utf-8', 'date': 'Sun, 27 Oct 2024 00:06:14 GMT'} -+-+- b'container not found ("default")'
...
K8sError: Error during: Execute command in pod. {"pod": "agent-env-a7mgqcsr-default-0", ...

With timestamps

2024-10-27 00:06:14,802 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-a7mgqcsr-default-0", ...
2024-10-27 00:06:15,115 - ERROR - K8S: Error during: Execute command in pod. {"cause": "(0)\nReason: Handshake status 500 Internal Server Error -+-+- {'content-length': '31', 'content-type': 'text/plain; charset=utf-8', 'date': 'Sun, 27 Oct 2024 00:06:14 GMT'} -+-+- b'container not found (\"default\")'\n", ...

Cluster events (note "SandboxChanged")

2024-10-26T23:45:04Z   Normal    agent-env-a7mgqcsr-default-0                  Scheduled               Successfully assigned agent/agent-env-a7mgqcsr-default-0 to ip-192-168-151-2.eu-west-2.compute.internal
2024-10-26T23:45:04Z   Normal    agent-env-a7mgqcsr-default                    SuccessfulCreate        create Pod agent-env-a7mgqcsr-default-0 in StatefulSet agent-env-a7mgqcsr-default successful
2024-10-26T23:45:05Z   Normal    agent-env-a7mgqcsr-default-0                  Created                 Created container resolve-coredns-ip
2024-10-26T23:45:05Z   Normal    agent-env-a7mgqcsr-default-0                  Pulled                  Container image "toolbelt/dig:2024-09-23" already present on machine
2024-10-26T23:45:05Z   Normal    agent-env-a7mgqcsr-default-0                  Started                 Started container resolve-coredns-ip
2024-10-26T23:45:06Z   Normal    agent-env-a7mgqcsr-default-0                  Pulled                  Container image "python:3.12-bookworm" already present on machine
2024-10-26T23:45:07Z   Normal    agent-env-a7mgqcsr-default-0                  Started                 Started container default
2024-10-26T23:45:07Z   Normal    agent-env-a7mgqcsr-default-0                  Created                 Created container default
2024-10-27T00:06:08Z   Normal    agent-env-a7mgqcsr-default-0                  SandboxChanged          Pod sandbox changed, it will be killed and re-created.
2024-10-27T00:06:13Z   Normal    agent-env-a7mgqcsr-default-0                  Created                 Created container resolve-coredns-ip
2024-10-27T00:06:13Z   Normal    agent-env-a7mgqcsr-default-0                  Pulled                  Container image "toolbelt/dig:2024-09-23" already present on machine
2024-10-27T00:06:14Z   Normal    agent-env-a7mgqcsr-default-0                  Started                 Started container resolve-coredns-ip
2024-10-27T00:06:15Z   Normal    agent-env-a7mgqcsr-default-0                  Pulled                  Container image "python:3.12-bookworm" already present on machine
2024-10-27T00:06:15Z   Normal    agent-env-a7mgqcsr-default-0                  Created                 Created container default
2024-10-27T00:06:15Z   Normal    agent-env-a7mgqcsr-default-0                  Started                 Started container default
2024-10-27T00:06:17Z   Normal    agent-env-a7mgqcsr-default-0                  Killing                 Stopping container default

---

06 Nov 2024

│ redacted/.venv/lib/python3.12/site-packages/websocket/_handshake.py:150 in _get_resp_headers    │
│                                                                                                                      │
│   147 │   │   │   )  # read the body of the HTTP error message response and include it in the                        │
│   148 │   │   else:                                                                                                  │
│   149 │   │   │   response_body = None                                                                               │
│ > 150 │   │   raise WebSocketBadStatusException(                                                                     │
│   151 │   │   │   f"Handshake status {status} {status_message} -+-+- {resp_headers} -+-+- {res                       │
│   152 │   │   │   status,                                                                                            │
│   153 │   │   │   status_message,                                                                                    │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
WebSocketBadStatusException: Handshake status 500 Internal Server Error -+-+- {'content-length': '31', 'content-type': 'text/plain; charset=utf-8', 'date': 'Tue, 05 Nov 2024 23:15:49 GMT'} -+-+- b'container not found ("default")'
...
│ redacted/.venv/lib/python3.12/site-packages/kubernetes/stream/ws_client.py:538 in               │
│ websocket_call                                                                                                       │
│                                                                                                                      │
│   535 │   │   else:                                                                                                  │
│   536 │   │   │   return WSResponse('%s' % ''.join(all))                                                             │
│   537 │   except (Exception, KeyboardInterrupt, SystemExit) as e:                                                    │
│ > 538 │   │   raise ApiException(status=0, reason=str(e))                                                            │
│   539                                                                                                                │
│   540                                                                                                                │
│   541 def portforward_call(configuration, _method, url, **kwargs):                                                   │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
ApiException: (0)
Reason: Handshake status 500 Internal Server Error -+-+- {'content-length': '31', 'content-type': 'text/plain; charset=utf-8', 'date': 'Tue, 05 Nov 2024 23:15:49 GMT'} -+-+- b'container not found ("default")'
...
K8sError: Error during: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0", ...

2024-11-05 22:28:53,161 - SANDBOX - K8S: Starting: Write file to pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:28:53,320 - SANDBOX - K8S: Completed: Write file to pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:29:49,475 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:29:49,693 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:29:49,693 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:29:49,802 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:30:29,761 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:30:54,481 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:30:54,481 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:30:54,588 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:31:23,680 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:31:26,082 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:31:26,082 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:31:26,204 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:32:31,732 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:32:31,976 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:33:22,228 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:33:22,633 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:34:08,098 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:34:08,285 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:34:52,291 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:34:52,587 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:34:52,587 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:34:52,731 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:35:52,309 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:36:06,006 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:36:06,006 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:36:06,116 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:37:14,616 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:37:14,772 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:37:14,773 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:37:14,900 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:39:34,692 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:39:34,884 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:40:39,403 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:40:39,624 - SANDBOX - K8S: Error during: Execute command in pod. {"cause": "'utf-8' ...
2024-11-05 22:42:07,421 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:42:07,722 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:42:07,722 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:42:07,859 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:43:32,065 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:43:32,350 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:43:32,350 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:43:32,733 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:45:06,100 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:45:06,709 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:45:06,709 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:45:06,834 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:46:01,742 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:46:02,621 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:46:02,621 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:46:02,791 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:47:30,587 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:47:30,767 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:48:29,348 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:48:29,485 - SANDBOX - K8S: Error during: Execute command in pod. {"cause": "'utf-8' ...
2024-11-05 22:50:08,606 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:50:08,999 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:50:08,999 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:50:09,321 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:51:44,178 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:51:44,363 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=False, returncode ...
2024-11-05 22:51:44,363 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:51:44,480 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:53:17,418 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:53:17,571 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:53:17,571 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:53:17,705 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:54:56,659 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:54:56,986 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:56:11,922 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:56:13,731 - SANDBOX - K8S: Completed: Execute command in pod. {"result": "ExecResult(success=True, returncode ...
2024-11-05 22:56:13,731 - SANDBOX - K8S: Starting: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:56:13,871 - SANDBOX - K8S: Completed: Read file from pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 22:57:57,959 - SANDBOX - K8S: Starting: Execute command in pod. {"pod": "agent-env-akqvhzlv-default-0 ...
2024-11-05 23:15:49,220 - ERROR - K8S: Error during: Execute command in pod. {"cause": "(0)\nReason ...

2024-11-05T22:28:42Z   Warning   agent-env-akqvhzlv-default-0                  FailedScheduling        0/46 nodes are available: 2 node(s) had untolerated taint {CriticalAddonsOnly: true}, 2 node(s) had untolerated taint {aisi.gov.uk/dev: true}, 2 node(s) had untolerated taint {aisi.gov.uk/devpods: true}, 2 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, 38 Insufficient memory. preemption: 0/46 nodes are available: 38 No preemption victims found for incoming pod, 8 Preemption is not helpful for scheduling.
2024-11-05T22:28:45Z   Normal    agent-env-akqvhzlv-default-0                  Scheduled               Successfully assigned agent/agent-env-akqvhzlv-default-0 to ip-192-168-174-135.eu-west-2.compute.internal
2024-11-05T22:28:46Z   Normal    agent-env-akqvhzlv-default-0                  Pulled                  Container image "redacted" already present on machine
2024-11-05T22:28:46Z   Normal    agent-env-akqvhzlv-default-0                  Started                 Started container resolve-coredns-ip
2024-11-05T22:28:46Z   Normal    agent-env-akqvhzlv-default-0                  Created                 Created container resolve-coredns-ip
2024-11-05T22:28:46Z   Normal    agent-env-akqvhzlv-default-0                  Pulled                  Container image "toolbelt/dig:2024-09-23" already present on machine
2024-11-05T22:28:47Z   Normal    agent-env-akqvhzlv-default-0                  Started                 Started container default
2024-11-05T22:28:47Z   Normal    agent-env-akqvhzlv-default-0                  Created                 Created container default
2024-11-05T22:57:21Z   Warning   agent-env-akqvhzlv-default-0                  NodeNotReady            Node is not ready
2024-11-05T23:02:26Z   Normal    agent-env-akqvhzlv-default-0                  TaintManagerEviction    Marking for deletion Pod agent/agent-env-akqvhzlv-default-0
2024-11-05T23:02:27Z   Normal    agent-env-akqvhzlv-default                    SuccessfulCreate        create Pod agent-env-akqvhzlv-default-0 in StatefulSet agent-env-akqvhzlv-default successful
2024-11-05T23:07:09Z   Warning   agent-env-akqvhzlv-default-0                  FailedScheduling        0/46 nodes are available: 2 node(s) had untolerated taint {CriticalAddonsOnly: true}, 2 node(s) had untolerated taint {aisi.gov.uk/dev: true}, 2 node(s) had untolerated taint {aisi.gov.uk/devpods: true}, 36 Insufficient memory, 4 node(s) had untolerated taint {node.kubernetes.io/unreachable: }. preemption: 0/46 nodes are available: 10 Preemption is not helpful for scheduling, 36 No preemption victims found for incoming pod.
2024-11-05T23:07:19Z   Normal    agent-env-akqvhzlv-default-0                  Scheduled               Successfully assigned agent/agent-env-akqvhzlv-default-0 to ip-192-168-189-169.eu-west-2.compute.internal
2024-11-05T23:07:20Z   Normal    agent-env-akqvhzlv-default-0                  Pulled                  Container image "toolbelt/dig:2024-09-23" already present on machine
2024-11-05T23:07:20Z   Normal    agent-env-akqvhzlv-default-0                  Created                 Created container resolve-coredns-ip
2024-11-05T23:07:21Z   Normal    agent-env-akqvhzlv-default-0                  Created                 Created container default
2024-11-05T23:07:21Z   Normal    agent-env-akqvhzlv-default-0                  Pulled                  Container image "redacted" already present on machine
2024-11-05T23:07:21Z   Normal    agent-env-akqvhzlv-default-0                  Started                 Started container resolve-coredns-ip
2024-11-05T23:07:22Z   Normal    agent-env-akqvhzlv-default-0                  Started                 Started container default
2024-11-05T23:15:49Z   Normal    agent-env-akqvhzlv-default-0                  SandboxChanged          Pod sandbox changed, it will be killed and re-created.
2024-11-05T23:15:50Z   Normal    agent-env-akqvhzlv-default-0                  Killing                 Stopping container default
2024-11-05T23:15:51Z   Normal    agent-env-akqvhzlv-default-0                  Pulled                  Container image "toolbelt/dig:2024-09-23" already present on machine
2024-11-05T23:15:51Z   Normal    agent-env-akqvhzlv-default-0                  Created                 Created container resolve-coredns-ip
2024-11-05T23:15:51Z   Normal    agent-env-akqvhzlv-default-0                  Started                 Started container resolve-coredns-ip
2024-11-05T23:15:52Z   Normal    agent-env-akqvhzlv-default-0                  Killing                 Stopping container resolve-coredns-ip