Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nodes get stuck with registration message, can't switch to log screens #2500

Open
scottyeager opened this issue Nov 25, 2024 · 1 comment
Open

Nodes get stuck with registration message, can't switch to log screens #2500

scottyeager opened this issue Nov 25, 2024 · 1 comment

Comments

@scottyeager
Copy link

We've seen a couple instances now where nodes are getting stuck in states where the key combos to switch to the log screens (alt-f1,f2) don't work. In both cases the node console displayed a different message suggesting that registration was still in progress. In at least one case, the node was actually functioning normally by all other measures (uptime reporting and logs submitted to our logging system).

Here's a screenshot from case one. This was a new node and the farmer reported that it always returned to this screen after reboot:

image

Here's a screenshot from the second case, node id 6862. Here I was able to query the node logs too which I'll include below:

image

[+] noded: 2024-11-25T19:56:27Z info utc time from tfchain: 2024-11-25 19:56:25 +0000 UTC
[+] noded: 2024-11-25T19:56:27Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:56:07.453024Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:55:59.361412Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] redis: 3934:M 25 Nov 2024 19:55:42.160 * Background saving terminated with success
[+] redis: 7736:C 25 Nov 2024 19:55:42.061 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
[+] redis: 7736:C 25 Nov 2024 19:55:42.061 * DB saved on disk
[+] redis: 3934:M 25 Nov 2024 19:55:42.059 * Background saving started by pid 7736
[+] redis: 3934:M 25 Nov 2024 19:55:42.058 * 100 changes in 300 seconds. Saving...
[+] noded: 2024-11-25T19:55:27Z info utc time from tfchain: 2024-11-25 19:55:24 +0000 UTC
[+] noded: 2024-11-25T19:55:26Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:54:26Z info utc time from tfchain: 2024-11-25 19:54:24 +0000 UTC
[+] noded: 2024-11-25T19:54:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:53:47.929145Z ERROR mycelium::peer_manager: Couldn't connect to endpoint, turn on debug logging for more details endpoint.address=5.223.43.251:9651 endpoint.proto=Tcp
[+] mycelium: 2024-11-25T19:53:43.292085Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.292083Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.292072Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.292072Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.292048Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.292058Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.292054Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.292035Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.292031Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291982Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291946Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291941Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291925Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291907Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291897Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291892Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291879Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291866Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291835Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291827Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291817Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291805Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291779Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.291778Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:53:43.206170Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] noded: 2024-11-25T19:53:26Z info utc time from tfchain: 2024-11-25 19:53:24 +0000 UTC
[+] noded: 2024-11-25T19:53:26Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:52:26Z info utc time from tfchain: 2024-11-25 19:52:24 +0000 UTC
[+] noded: 2024-11-25T19:52:26Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:51:26Z info utc time from tfchain: 2024-11-25 19:51:24 +0000 UTC
[+] noded: 2024-11-25T19:51:26Z info running NTP check against tfchain
[+] redis: 3934:M 25 Nov 2024 19:50:41.165 * Background saving terminated with success
[+] redis: 7445:C 25 Nov 2024 19:50:41.066 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
[+] redis: 7445:C 25 Nov 2024 19:50:41.066 * DB saved on disk
[+] redis: 3934:M 25 Nov 2024 19:50:41.064 * Background saving started by pid 7445
[+] redis: 3934:M 25 Nov 2024 19:50:41.064 * 100 changes in 300 seconds. Saving...
[+] noded: 2024-11-25T19:50:26Z info utc time from tfchain: 2024-11-25 19:50:24 +0000 UTC
[+] noded: 2024-11-25T19:50:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:50:12.815921Z ERROR mycelium::peer_manager: Couldn't connect to endpoint, turn on debug logging for more details endpoint.address=136.243.47.186:9651 endpoint.proto=Tcp
[+] mycelium: 2024-11-25T19:50:08.759101Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.757270Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.725892Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693802Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693801Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693788Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693780Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693785Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693733Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693768Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693762Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693737Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693722Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693702Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693690Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693664Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693662Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693621Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.693617Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:50:08.624243Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:49:46.371139Z ERROR mycelium::peer_manager: Couldn't connect to endpoint, turn on debug logging for more details endpoint.address=5.78.122.16:9651 endpoint.proto=Tcp
[+] noded: 2024-11-25T19:49:26Z info utc time from tfchain: 2024-11-25 19:49:24 +0000 UTC
[+] noded: 2024-11-25T19:49:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:49:13.603089Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] mycelium: 2024-11-25T19:48:28.547160Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] noded: 2024-11-25T19:48:26Z info utc time from tfchain: 2024-11-25 19:48:24 +0000 UTC
[+] noded: 2024-11-25T19:48:26Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:48:23Z error failed health check. retrying error="failed health check" check=network retry-in=22267.74502
[+] noded: 2024-11-25T19:47:53Z error failed health check. retrying error="failed health check" check=network retry-in=25016.572254
[+] mycelium: 2024-11-25T19:47:39.550551Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:47:39.395135Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] noded: 2024-11-25T19:47:26Z info utc time from tfchain: 2024-11-25 19:47:24 +0000 UTC
[+] noded: 2024-11-25T19:47:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:47:18.915130Z ERROR mycelium::peer_manager: Couldn't connect to endpoint, turn on debug logging for more details endpoint.address=142.93.217.194:9651 endpoint.proto=Tcp
[+] noded: 2024-11-25T19:47:09Z error failed health check. retrying error="failed health check" check=network retry-in=39037.774522
[+] mycelium: 2024-11-25T19:47:03.604420Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:46:46.147123Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] noded: 2024-11-25T19:46:32Z error failed health check. retrying error="failed health check" check=network retry-in=31981.212211
[+] noded: 2024-11-25T19:46:26Z info utc time from tfchain: 2024-11-25 19:46:24 +0000 UTC
[+] noded: 2024-11-25T19:46:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:45:56.995161Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] mycelium: 2024-11-25T19:45:56.995137Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] noded: 2024-11-25T19:45:45Z error failed health check. retrying error="failed health check" check=network retry-in=42552.204515
[+] mycelium: 2024-11-25T19:45:44.958824Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] redis: 3934:M 25 Nov 2024 19:45:40.180 * Background saving terminated with success
[+] redis: 7143:C 25 Nov 2024 19:45:40.080 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
[+] redis: 7143:C 25 Nov 2024 19:45:40.080 * DB saved on disk
[+] redis: 3934:M 25 Nov 2024 19:45:40.078 * Background saving started by pid 7143
[+] redis: 3934:M 25 Nov 2024 19:45:40.077 * 100 changes in 300 seconds. Saving...
[+] noded: 2024-11-25T19:45:26Z info utc time from tfchain: 2024-11-25 19:45:24 +0000 UTC
[+] noded: 2024-11-25T19:45:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:45:07.843133Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] noded: 2024-11-25T19:45:05Z error failed health check. retrying error="failed health check" check=network retry-in=35219.625583
[+] noded: 2024-11-25T19:44:26Z info utc time from tfchain: 2024-11-25 19:44:24 +0000 UTC
[+] noded: 2024-11-25T19:44:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:44:10.499129Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] noded: 2024-11-25T19:43:26Z info utc time from tfchain: 2024-11-25 19:43:24 +0000 UTC
[+] noded: 2024-11-25T19:43:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:42:48.579132Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Operation timed out (os error 110)
[+] noded: 2024-11-25T19:42:26Z info utc time from tfchain: 2024-11-25 19:42:24 +0000 UTC
[+] noded: 2024-11-25T19:42:26Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:41:26Z info utc time from tfchain: 2024-11-25 19:41:24 +0000 UTC
[+] noded: 2024-11-25T19:41:26Z info running NTP check against tfchain
[+] redis: 3934:M 25 Nov 2024 19:40:39.193 * Background saving terminated with success
[+] redis: 6892:C 25 Nov 2024 19:40:39.093 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
[+] redis: 6892:C 25 Nov 2024 19:40:39.093 * DB saved on disk
[+] redis: 3934:M 25 Nov 2024 19:40:39.091 * Background saving started by pid 6892
[+] redis: 3934:M 25 Nov 2024 19:40:39.090 * 100 changes in 300 seconds. Saving...
[+] noded: 2024-11-25T19:40:26Z info utc time from tfchain: 2024-11-25 19:40:25 +0000 UTC
[+] noded: 2024-11-25T19:40:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:39:31.971129Z ERROR mycelium::peer_manager: Couldn't connect to endpoint, turn on debug logging for more details endpoint.address=142.93.217.194:9651 endpoint.proto=Tcp
[+] noded: 2024-11-25T19:39:26Z info utc time from tfchain: 2024-11-25 19:39:25 +0000 UTC
[+] noded: 2024-11-25T19:39:26Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:38:26Z info utc time from tfchain: 2024-11-25 19:38:24 +0000 UTC
[+] noded: 2024-11-25T19:38:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:38:06.202634Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] noded: 2024-11-25T19:37:26Z info utc time from tfchain: 2024-11-25 19:37:25 +0000 UTC
[+] noded: 2024-11-25T19:37:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:36:49.679055Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] noded: 2024-11-25T19:36:26Z info utc time from tfchain: 2024-11-25 19:36:25 +0000 UTC
[+] noded: 2024-11-25T19:36:26Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:36:23.555138Z ERROR mycelium::peer_manager: Couldn't connect to endpoint, turn on debug logging for more details endpoint.address=5.78.122.16:9651 endpoint.proto=Tcp
[+] redis: 3934:M 25 Nov 2024 19:35:38.109 * Background saving terminated with success
[+] redis: 6628:C 25 Nov 2024 19:35:38.010 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
[+] redis: 6628:C 25 Nov 2024 19:35:38.010 * DB saved on disk
[+] redis: 3934:M 25 Nov 2024 19:35:38.008 * Background saving started by pid 6628
[+] redis: 3934:M 25 Nov 2024 19:35:38.008 * 100 changes in 300 seconds. Saving...
[+] noded: 2024-11-25T19:35:26Z info utc time from tfchain: 2024-11-25 19:35:24 +0000 UTC
[+] noded: 2024-11-25T19:35:25Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:35:14.076713Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] mycelium: 2024-11-25T19:34:26.315033Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Broken pipe (os error 32)
[+] noded: 2024-11-25T19:34:25Z info utc time from tfchain: 2024-11-25 19:34:24 +0000 UTC
[+] noded: 2024-11-25T19:34:25Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:33:25Z info utc time from tfchain: 2024-11-25 19:33:24 +0000 UTC
[+] noded: 2024-11-25T19:33:25Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:32:25Z info utc time from tfchain: 2024-11-25 19:32:24 +0000 UTC
[+] noded: 2024-11-25T19:32:25Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:31:25Z info utc time from tfchain: 2024-11-25 19:31:24 +0000 UTC
[+] noded: 2024-11-25T19:31:25Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:30:52Z error failed health check. retrying error="failed health check" check=network retry-in=43353.923346
[+] redis: 3934:M 25 Nov 2024 19:30:37.121 * Background saving terminated with success
[+] redis: 6354:C 25 Nov 2024 19:30:37.022 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
[+] redis: 6354:C 25 Nov 2024 19:30:37.022 * DB saved on disk
[+] redis: 3934:M 25 Nov 2024 19:30:37.020 * Background saving started by pid 6354
[+] redis: 3934:M 25 Nov 2024 19:30:37.019 * 100 changes in 300 seconds. Saving...
[+] noded: 2024-11-25T19:30:25Z info utc time from tfchain: 2024-11-25 19:30:24 +0000 UTC
[+] noded: 2024-11-25T19:30:25Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:30:05Z error failed health check. retrying error="failed health check" check=network retry-in=42208.854096
[+] noded: 2024-11-25T19:29:25Z info utc time from tfchain: 2024-11-25 19:29:24 +0000 UTC
[+] noded: 2024-11-25T19:29:25Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:28:36.611243Z ERROR mycelium::peer_manager: Couldn't connect to endpoint, turn on debug logging for more details endpoint.address=5.78.122.16:9651 endpoint.proto=Tcp
[+] mycelium: 2024-11-25T19:28:36.611177Z ERROR mycelium::peer_manager: Couldn't connect to endpoint, turn on debug logging for more details endpoint.address=142.93.217.194:9651 endpoint.proto=Tcp
[+] noded: 2024-11-25T19:28:25Z info utc time from tfchain: 2024-11-25 19:28:24 +0000 UTC
[+] noded: 2024-11-25T19:28:25Z info running NTP check against tfchain
[-] ntp: ntpd: reply from 81.172.225.142: delay 0.039543 is too high, ignoring
[-] ntp: ntpd: reply from 178.239.19.57: delay 0.196396 is too high, ignoring
[+] noded: 2024-11-25T19:27:36Z error failed health check. retrying error="failed health check" check=network retry-in=41000.998011
[+] noded: 2024-11-25T19:27:25Z info utc time from tfchain: 2024-11-25 19:27:24 +0000 UTC
[+] noded: 2024-11-25T19:27:25Z info running NTP check against tfchain
[+] mycelium: 2024-11-25T19:27:09.698488Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Broken pipe (os error 32)
[+] mycelium: 2024-11-25T19:27:08.973058Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Broken pipe (os error 32)
[+] mycelium: 2024-11-25T19:27:08.688747Z ERROR mycelium::peer: Failed to flush buffered peer connection control packets: Connection reset by peer (os error 104)
[+] noded: 2024-11-25T19:27:04Z error failed health check. retrying error="failed health check" check=network retry-in=26609.921505
[+] noded: 2024-11-25T19:26:44Z warn skipped, there is a node with less ID available
[+] powerd: 2024-11-25T19:26:42Z info setting node power state state=true
[+] powerd: 2024-11-25T19:26:42Z info listening for power events
[+] powerd: 2024-11-25T19:26:42Z info node uptime hash hash=0x4a22db203db6d0f577c06cac20638bc08ca96c636abdf1059e74e3121a5dcc90
[+] noded: 2024-11-25T19:26:37Z info setting node public config
[+] powerd: 2024-11-25T19:26:37Z info enabling wol on interface nic=enp5s0
[+] powerd: 2024-11-25T19:26:37Z error node can not reach grid services
[+] powerd: 2024-11-25T19:26:37Z info node address address=5FiHf9pCyjXn24FicHZYfax51C6KXjzfVxPhgipFWVnXUvkJ
[+] noded: 2024-11-25T19:26:37Z info node address address=5FiHf9pCyjXn24FicHZYfax51C6KXjzfVxPhgipFWVnXUvkJ
[-] noded: 2024/11/25 19:26:37 Connecting to wss://tfchain.02.grid.tf...
[+] noded: 2024-11-25T19:26:37Z info node has been registered twin=12529
[+] noded: 2024-11-25T19:26:37Z info node registered node=6862 twin=12529
[+] powerd: 2024-11-25T19:26:35Z fatal exiting error="failed to get node id: registration is still in progress"
[+] powerd: 2024-11-25T19:26:33Z fatal exiting error="failed to get node id: registration is still in progress"
[+] noded: 2024-11-25T19:26:33Z warn the node isn't ready yet err="registration is still in progress" sleep=4.34292667s
[+] noded: 2024-11-25T19:26:31Z error failed to get node id error="registration is still in progress"
[+] powerd: 2024-11-25T19:26:31Z fatal exiting error="failed to get node id: registration is still in progress"
[+] provisiond: 2024-11-25T19:26:30Z warn failed to disable frame-buffer error="write /sys/bus/platform/drivers/efi-framebuffer/unbind: no such device"
[+] provisiond: 2024-11-25T19:26:30Z error error="should find only one namespace with prefix \"zdb-ns-\", found 0"
[+] noded: 2024-11-25T19:26:30Z info registering node on blockchain
[+] noded: 2024-11-25T19:26:30Z info start registration of the node id=BrvGu7kGYNu3yE7TbFnsi6jWZoKhLS3XQGRSH8p2HayB
[+] gateway: 2024-11-25T19:26:30Z info starting gateway module broker=unix:///var/run/redis.sock worker nr=1
[+] gateway: 2024-11-25T19:26:30Z error gateway is not supported error="gateway is not supported on this node: no public configuration"
[+] noded: 2024-11-25T19:26:30Z error failed health check. retrying error="failed health check" check=network retry-in=29852.723966
[+] noded: 2024-11-25T19:26:29Z warn the node isn't ready yet err="registration is still in progress" sleep=3.244635395s
[+] powerd: 2024-11-25T19:26:29Z fatal exiting error="failed to get node id: registration is still in progress"
[+] noded: 2024-11-25T19:26:27Z warn the node isn't ready yet err="registration is still in progress" sleep=2.015705094s
[+] powerd: 2024-11-25T19:26:27Z fatal exiting error="failed to get node id: registration is still in progress"
[+] provisiond: 2024-11-25T19:26:27Z info starting provision module broker=unix:///var/run/redis.sock
[+] provisiond: 2024-11-25T19:26:27Z info setting contracts used capacity done
[+] provisiond: 2024-11-25T19:26:27Z info setting used capacity by contracts contracts=0
[+] noded: 2024-11-25T19:26:26Z warn the node isn't ready yet err="registration is still in progress" sleep=1.228955407s
[+] flistd: 2024-11-25T19:26:26Z info request to mount flist storage= url=https://hub.grid.tf/tf-autobuilder/traefik-2.9.9.flist
[+] flistd: 2024-11-25T19:26:26Z info request to mount flist: {ReadOnly:true Limit:0 Storage: PersistedVolume:} name=traefik:4e13bd44ea4c7d0c8fb6502d53c4032f storage= url=https://hub.grid.tf/tf-autobuilder/traefik-2.9.9.flist
[+] noded: 2024-11-25T19:26:25Z warn the node isn't ready yet err="registration is still in progress" sleep=1.038066705s
[+] noded: 2024-11-25T19:26:25Z info utc time from tfchain: 2024-11-25 19:26:25 +0000 UTC
[+] vmd: 2024-11-25T19:26:25Z info starting vmd module broker=unix:///var/run/redis.sock worker nr=1
[+] powerd: 2024-11-25T19:26:25Z fatal exiting error="failed to get node id: registration is still in progress"
[+] noded: 2024-11-25T19:26:25Z warn the node isn't ready yet err="registration is still in progress" sleep=615.114902ms
[+] noded: 2024-11-25T19:26:25Z info running NTP check against tfchain
[+] noded: 2024-11-25T19:26:25Z info start perf scheduler
[+] noded: 2024-11-25T19:26:25Z info found GPU device="GP106GL [Quadro P2000]" vendor="NVIDIA Corporation"
[+] provisiond: 2024-11-25T19:26:24Z info run integrity checks
[+] networkd/test: 
[+] networkd/test:   time: 2024-11-25T19:26:22.762470236Z
[+] networkd/test: - state: free
[+] networkd/test: workers:
[+] networkd/test:   version: 0.0.1
[+] networkd/test: - name: network
[+] networkd/test: objects:
[+] networkd/test: ## Status for  network
[+] networkd: 2024-11-25T19:26:22Z info starting networkd module broker=unix:///var/run/redis.sock worker nr=1
[+] networkd: 2024-11-25T19:26:22Z info start zbus server
[+] mycelium: 2024-11-25T19:26:22.760868Z ERROR mycelium::peer_manager: Could not send multicast discovery beacon on interface iface=7 err=Address not available (os error 99)
[+] networkd/test: 2024-11-25T19:26:22Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:26:22Z error failed to get status for module error="context deadline exceeded" module=network
[+] networkd: 2024-11-25T19:26:22Z info filtering out peers from ranges: [{fd00::1 ffffffffffffffff0000000000000000} {2a0d:6e00:11bb:1:a85c:fdff:fee0:a25 ffffffffffffffff0000000000000000} {300:bd42:de22:ea41::1 ffffffffffffffff0000000000000000} {200:bd42:de22:ea41:45f5:8956:a7af:b981 fe000000000000000000000000000000}]
[+] yggdrasil: 2024/11/25 19:26:21 Your IPv6 subnet is 300:bd42:de22:ea41::/64
[+] yggdrasil: 2024/11/25 19:26:21 Your IPv6 address is 200:bd42:de22:ea41:45f5:8956:a7af:b981
[+] yggdrasil: 2024/11/25 19:26:21 Your public key is a15e90ee8adf5d053b54ac28233f2e4401c67165e9917d0d0d9e0e51d9f9d70e
[+] yggdrasil/test: 2024/11/25 19:26:21 Fatal error: dial unix /var/run/yggdrasil/yggdrasil.sock: connect: no such file or directory
[+] yggdrasil/test: 2024/11/25 19:26:21 Connecting to UNIX socket /var/run/yggdrasil/yggdrasil.sock
[+] yggdrasil/test: 2024/11/25 19:26:21 Falling back to platform default unix:///var/run/yggdrasil/yggdrasil.sock
[+] yggdrasil/test: 2024/11/25 19:26:21 Can't open config file from default location /etc/yggdrasil/yggdrasil.conf
[+] networkd/test: ## Status for  network
[+] networkd: 2024-11-25T19:26:19Z error failed to lookup ip error="lookup jorropo.net on [2001:4860:4860::8888]:53: server misbehaving" url=tls://jorropo.net:12221?key=060f2d49c6a1a2066357ea06e58f5cff8c76a5c0cc513ceb2dab75c900fe183b&sni=jorropo.net
[+] networkd/test: 2024-11-25T19:26:17Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:26:17Z error failed to get status for module error="context deadline exceeded" module=network
[+] networkd/test: ## Status for  network
[+] networkd/test: 2024-11-25T19:26:12Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:26:12Z error failed to get status for module error="context deadline exceeded" module=network
[+] networkd/test: ## Status for  network
[+] networkd/test: 2024-11-25T19:26:07Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:26:07Z error failed to get status for module error="context deadline exceeded" module=network
[-] dhcp-npub4: npub4: adding default route via 192.168.1.1
[-] dhcp-npub4: npub4: adding route to 192.168.1.0/24
[-] dhcp-npub4: npub4: leased 192.168.1.37 for 86400 seconds
[+] networkd/test: ## Status for  network
[+] networkd/test: 2024-11-25T19:26:02Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:26:02Z error failed to get status for module error="context deadline exceeded" module=network
[-] dhcp-npub4: npub4: probing address 192.168.1.37/24
[-] dhcp-npub4: npub4: offered 192.168.1.37 from 192.168.1.1
[-] dhcp-npub4: npub4: soliciting a DHCP lease
[+] networkd/test: ## Status for  network
[-] dhcp-npub4: npub4: soliciting an IPv6 router
[-] dhcp-npub4: ipv6_addaddr1: Permission denied
[-] dhcp-npub4: npub4: adding address fe80::d1e3:87ef:532:d88d
[-] dhcp-npub4: npub4: IAID 9e:2a:fe:26
[-] dhcp-npub4: DUID 00:04:c4:4d:c8:c2:d7:fe:11:e7:9c:43:bc:00:00:4c:00:00
[+] networkd: 2024-11-25T19:25:59Z info filtering out peers from ranges: [{fd00::1 ffffffffffffffff0000000000000000} {2a0d:6e00:11bb:1:a85c:fdff:fee0:a25 ffffffffffffffff0000000000000000}]
[+] networkd: 2024-11-25T19:25:59Z info create dhcp-npub4 zinit service
[+] networkd: 2024-11-25T19:25:59Z info wait for slaac to give ipv6
[+] api-gateway: 2024-11-25T19:25:58Z info starting api-gateway module broker=unix:///var/run/redis.sock worker nr=1
[+] api-gateway: 2024-11-25T19:25:57Z info starting peer session= twin=12529
[+] networkd: 2024-11-25T19:25:57Z info wait for slaac to give ipv6
[+] networkd/test: 2024-11-25T19:25:57Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:25:57Z error failed to get status for module error="context deadline exceeded" module=network
[+] networkd: 2024-11-25T19:25:57Z info wait for slaac to give ipv6
[+] networkd: 2024-11-25T19:25:56Z info wait for slaac to give ipv6
[+] networkd: 2024-11-25T19:25:56Z info link didn't match filter criteria (PhysicalFilter), skip testing interface=br-pub type=bridge
[+] networkd: 2024-11-25T19:25:56Z info filtering interface interface=br-pub type=bridge
[+] networkd: 2024-11-25T19:25:56Z info link didn't match filter criteria (PhysicalFilter), skip testing interface=zos type=bridge
[+] networkd: 2024-11-25T19:25:56Z info filtering interface interface=zos type=bridge
[+] networkd: 2024-11-25T19:25:56Z info link didn't match filter criteria (PluggedFilter), skip testing interface=enp0s25 type=device
[-] api-gateway: 2024/11/25 19:25:55 Connecting to wss://02.tfchain.grid.tf/...
[+] networkd/test: ## Status for  network
[-] api-gateway: 2024/11/25 19:25:53 Connecting to wss://03.tfchain.grid.tf/...
[+] networkd/test: 2024-11-25T19:25:52Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:25:52Z error failed to get status for module error="context deadline exceeded" module=network
[-] api-gateway: 2024/11/25 19:25:50 Connecting to wss://tfchain.02.grid.tf...
[+] networkd/test: ## Status for  network
[+] networkd/test: 2024-11-25T19:25:46Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:25:46Z error failed to get status for module error="context deadline exceeded" module=network
[+] networkd: 2024-11-25T19:25:46Z info check if interface has a cable plugged in iface=enp0s25
[+] networkd: 2024-11-25T19:25:46Z info filtering interface interface=enp0s25 type=device
[+] networkd: 2024-11-25T19:25:46Z info link didn't match filter criteria (NotAttachedFilter), skip testing interface=enp5s0 type=device
[+] networkd: 2024-11-25T19:25:46Z info filtering interface interface=enp5s0 type=device
[+] networkd: 2024-11-25T19:25:46Z info filtering interface interface=lo type=device
[-] api-gateway: 2024/11/25 19:25:46 Connecting to wss://tfchain.grid.tf/...
[+] api-gateway: 2024-11-25T19:25:46Z info node address address=5FiHf9pCyjXn24FicHZYfax51C6KXjzfVxPhgipFWVnXUvkJ
[+] networkd: 2024-11-25T19:25:46Z info ensuring existing host nft rules
[+] networkd/test: ## Status for  network
[+] networkd/test: 2024-11-25T19:25:41Z fatal exiting error="context deadline exceeded"
[+] networkd/test: 2024-11-25T19:25:41Z error failed to get status for module error="context deadline exceeded" module=network
[+] storaged: 2024-11-25T19:25:41Z info starting storaged module broker=unix:///var/run/redis.sock worker nr=1
[+] storaged: 2024-11-25T19:25:41Z info storage initialization complete
[+] identityd: 2024-11-25T19:25:39Z info checking for update after=1h4m0s
[+] identityd: 2024-11-25T19:25:38Z info detecting boot method
[+] flistd: 2024-11-25T19:25:38Z info starting flist module broker=unix:///var/run/redis.sock worker nr=1
[+] qsfsd: 2024-11-25T19:25:38Z info starting qsfsd module broker=unix:///var/run/redis.sock worker nr=1
[+] contd: 2024-11-25T19:25:38Z info starting containerd module broker=unix:///var/run/redis.sock worker nr=1
[+] contd: 2024-11-25T19:25:38Z info wait for shim-logs binary to be available
[+] networkd: 2024-11-25T19:25:38Z info wait for yggdrasil binary to be available
[+] identityd/test: 
[+] identityd/test:   action: '[[email protected]].Farm()'
[+] identityd/test:   time: 2024-11-25T19:25:38.685819163Z
[+] identityd/test: - state: busy
[+] identityd/test: workers:
[+] identityd/test:   version: 0.0.1
[+] identityd/test: - name: manager
[+] identityd/test:   version: 0.0.1
[+] identityd/test: - name: monitor
[+] identityd/test: objects:
[+] identityd/test: ## Status for  identityd
[+] networkd/test: ## Status for  network
[-] identityd: 2024/11/25 19:25:38 Connecting to wss://tfchain.grid.tf/...
[+] identityd: 2024-11-25T19:25:38Z info current version=3.12.9
[+] identityd: 2024-11-25T19:25:38Z info farmer identified farmer_id=3229 orphan=false
[+] identityd: 2024-11-25T19:25:38Z info node identity loaded identity=BrvGu7kGYNu3yE7TbFnsi6jWZoKhLS3XQGRSH8p2HayB
[+] identityd: 2024-11-25T19:25:38Z info key store loaded kind=file-store
[+] storaged/test: /var/cache is a mountpoint
[+] storaged: 2024-11-25T19:25:37Z info Setting up cache
[-] containerd: time="2024-11-25T19:25:36.805946985Z" level=debug msg="garbage collected" d="134.182µs"
[+] storaged: 2024-11-25T19:25:36Z info pool creations completed broken-devices=0 broken-pools=0 hdd-pools=0 ssd-pools=2
[+] storaged: 2024-11-25T19:25:36Z info Initializing storage module
[+] containerd/test:   UUID: 39e01721-e3f7-4f87-bbef-1af776303cd4
[+] containerd/test:   Revision: .m
[+] containerd/test:   Version:  
[+] containerd/test: Server:
[-] containerd: time="2024-11-25T19:25:36.706532818Z" level=info msg="containerd successfully booted in 0.024165s"
[-] containerd: time="2024-11-25T19:25:36.706520028Z" level=debug msg="sd notification" error="<nil>" notified=false state="READY=1"
[-] containerd: time="2024-11-25T19:25:36.706499709Z" level=info msg=serving... address=/run/containerd/containerd.sock
[-] containerd: time="2024-11-25T19:25:36.706449932Z" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
[-] containerd: time="2024-11-25T19:25:36.706200830Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.706141558Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
[-] containerd: time="2024-11-25T19:25:36.706103236Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
[-] containerd: time="2024-11-25T19:25:36.706084090Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
[-] containerd: time="2024-11-25T19:25:36.706072652Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
[-] containerd: time="2024-11-25T19:25:36.706055669Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
[-] containerd: time="2024-11-25T19:25:36.706044329Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.706021878Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.706008742Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705995659Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705980981Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705966274Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705941242Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705928737Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705915236Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705901618Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705888217Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
[-] containerd: time="2024-11-25T19:25:36.705876405Z" level=debug msg="No RDT config file specified, RDT not configured"
[-] containerd: time="2024-11-25T19:25:36.705861163Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705847611Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705835241Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705821604Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705809482Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705794794Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705770438Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705756722Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705735016Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
[-] containerd: time="2024-11-25T19:25:36.705397877Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
[-] containerd: time="2024-11-25T19:25:36.705331970Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
[-] containerd: time="2024-11-25T19:25:36.705257380Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
[-] containerd: time="2024-11-25T19:25:36.705183699Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
[-] containerd: time="2024-11-25T19:25:36.705125820Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
[-] containerd: time="2024-11-25T19:25:36.705112667Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
[-] containerd: time="2024-11-25T19:25:36.705085829Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
[-] containerd: time="2024-11-25T19:25:36.704850326Z" level=info msg="metadata content store policy set" policy=shared
[-] containerd: time="2024-11-25T19:25:36.704840066Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
[-] containerd: time="2024-11-25T19:25:36.704811229Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
[-] containerd: time="2024-11-25T19:25:36.704793927Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.704676657Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.704517161Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.704478093Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.704468753Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
[-] containerd: time="2024-11-25T19:25:36.704455101Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.704437815Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/containerd/io.containerd.snapshotter.v1.btrfs (tmpfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.704242856Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.704197523Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
[-] containerd: time="2024-11-25T19:25:36.704164184Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/6.8.8-Zero-OS-23ebebd9f6\\n\"): skip plugin" type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.702644699Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
[-] containerd: time="2024-11-25T19:25:36.683301769Z" level=info msg="starting containerd" revision=.m version=
[-] containerd: time="2024-11-25T19:25:36Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, see https://github.com/containerd/containerd/blob/main/docs/PLUGINS.md#version-header"
[+] containerd/test: 
[+] containerd/test:   Go version: go1.23.3
[+] containerd/test:   Revision: .m
[+] containerd/test:   Version:  
[+] containerd/test: Client:
[+] storaged/test: /var/cache is not a mountpoint
[+] sysctl: fs.inotify.max_user_watches = 524288
[+] sysctl: fs.inotify.max_user_instances = 8192
[-] node-ready: rmmod: ERROR: Module libata is in use by: libahci ahci
[-] node-ready: rmmod: ERROR: Module ata_generic is not currently loaded
[-] node-ready: rmmod: ERROR: Module pata_acpi is not currently loaded
[+] redis: 3934:M 25 Nov 2024 19:25:36.386 * Ready to accept connections unix
[+] redis: 3934:M 25 Nov 2024 19:25:36.386 * Server initialized
[-] node-ready: rmmod: ERROR: Module ata_piix is not currently loaded
[+] redis: 3934:M 25 Nov 2024 19:25:36.384 * Running mode=standalone, port=0.
[+] redis: 3934:M 25 Nov 2024 19:25:36.384 * monotonic clock: POSIX clock_gettime
[+] redis: 3934:M 25 Nov 2024 19:25:36.384 * Increased maximum number of open files to 10032 (it was originally set to 1024).
[+] redis: 3934:C 25 Nov 2024 19:25:36.383 * Configuration loaded
[+] redis: 3934:C 25 Nov 2024 19:25:36.383 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=3934, just started
[+] redis: 3934:C 25 Nov 2024 19:25:36.383 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
[+] redis: 3934:C 25 Nov 2024 19:25:36.383 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[-] ntp: ntpd: reply from 178.239.19.57: delay 0.108954 is too high, ignoring
[-] ntp: ntpd: reply from 185.224.145.68: delay 0.139872 is too high, ignoring
[-] ntp: ntpd: reply from 213.109.127.82: delay 0.062830 is too high, ignoring
[-] ntp: ntpd: timed out waiting for 178.239.19.57, reach 0xfe, next query in 65s
[-] ntp: ntpd: reply from 213.109.127.82: delay 0.145956 is too high, ignoring
[-] ntp: ntpd: reply from 178.239.19.57: delay 0.274261 is too high, ignoring
[-] ntp: ntpd: reply from 81.172.225.142: delay 0.274204 is too high, ignoring
[-] ntp: ntpd: reply from 213.109.127.82: delay 0.284504 is too high, ignoring
[-] ntp: ntpd: reply from 81.172.225.142: delay 0.284474 is too high, ignoring
[-] ntp: ntpd: reply from 178.239.19.57: delay 0.284487 is too high, ignoring
[-] ntp: ntpd: reply from 185.224.145.68: delay 0.216951 is too high, ignoring
[-] ntp: ntpd: reply from 213.109.127.82: delay 0.094932 is too high, ignoring
[-] ntp: ntpd: reply from 178.239.19.57: delay 0.084949 is too high, ignoring
[-] ntp: ntpd: reply from 81.172.225.142: delay 0.084392 is too high, ignoring
[-] dhcp-zos: zos: adding default route via 192.168.1.1
[-] dhcp-zos: zos: adding route to 192.168.1.0/24
[-] dhcp-zos: zos: leased 192.168.1.31 for 86400 seconds
[-] dhcp-zos: zos: adding default route via fe80::18a1:38fc:770b:22ab
[-] dhcp-zos: zos: adding route to 2a0d:6e00:11bb:1::/64
[-] dhcp-zos: zos: adding address 2a0d:6e00:11bb:1:50ba:fc33:2c08:d0e6/64
[-] dhcp-zos: zos: Router Advertisement from fe80::18a1:38fc:770b:22ab
[-] dhcp-zos: zos: soliciting an IPv6 router
[-] dhcp-zos: zos: probing address 192.168.1.31/24
[-] dhcp-zos: zos: offered 192.168.1.31 from 192.168.1.1
[-] dhcp-zos: zos: soliciting a DHCP lease
[-] dhcp-zos: zos: IAID 82:5f:09:6b
[-] dhcp-zos: DUID 00:04:c4:4d:c8:c2:d7:fe:11:e7:9c:43:bc:00:00:4c:00:00
[-] dhcp-zos: zos: carrier acquired
[-] dhcp-zos: zos: waiting for carrier
[-] sshd: Server listening on 0.0.0.0 port 22.
[-] sshd: Server listening on :: port 22.
[+] sshd-setup: +----[SHA256]-----+
[+] sshd-setup: | Eo+o...o+ .     |
[+] sshd-setup: |o.o   +o+ = *    |
[+] sshd-setup: |oo=  o X.* * X   |
[+] sshd-setup: | =.=. +.= B * .  |
[+] sshd-setup: |o o +  oS  . .   |
[+] sshd-setup: |.  . +  ..       |
[+] sshd-setup: |      .          |
[+] sshd-setup: |                 |
[+] sshd-setup: |                 |
[+] sshd-setup: +--[ED25519 256]--+
[+] sshd-setup: The key's randomart image is:
[+] sshd-setup: SHA256:6c4DC3fkds5yF9OaId4xCzmjWLKrCtWlqbGcgOu/kMs root@zero-os
[+] sshd-setup: The key fingerprint is:
[+] sshd-setup: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub.
[+] sshd-setup: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key.
[+] sshd-setup: Generating public/private ed25519 key pair.
[+] sshd-setup: +----[SHA256]-----+
[+] sshd-setup: |          o+Oo*B*|
[+] sshd-setup: |         + = ***E|
[+] sshd-setup: |        = + ...O*|
[+] sshd-setup: |       + o . .+.=|
[+] sshd-setup: |      . S   o. + |
[+] sshd-setup: |       . .  o.o .|
[+] sshd-setup: |        .  . = o |
[+] sshd-setup: |            . + .|
[+] sshd-setup: |               . |
[+] sshd-setup: +---[ECDSA 521]---+
[+] sshd-setup: The key's randomart image is:
[+] sshd-setup: SHA256:Dke7M22Vu+mw+c6dHUaBtbhw/OuJ44cfltccjj+X124 root@zero-os
[+] sshd-setup: The key fingerprint is:
[+] sshd-setup: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub.
[+] sshd-setup: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key.
[+] sshd-setup: Generating public/private ecdsa key pair.
[+] sshd-setup: +----[SHA256]-----+
[+] sshd-setup: |       .+=O.o    |
[+] sshd-setup: |       ..X+B .   |
[+] sshd-setup: |       .=o* o    |
[+] sshd-setup: |       o.. +     |
[+] sshd-setup: |      ooS o .    |
[+] sshd-setup: |   . +.Eo  o o   |
[+] sshd-setup: |    +...  . = +  |
[+] sshd-setup: |     ... . = * + |
[+] sshd-setup: |      +=o . + o o|
[+] sshd-setup: +---[DSA 1024]----+
[+] sshd-setup: The key's randomart image is:
[+] sshd-setup: SHA256:8uMe5zFRnZWY2GgIxMqDVaEfux+zr2M73GlzQA23KjA root@zero-os
[+] sshd-setup: The key fingerprint is:
[+] sshd-setup: Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub.
[+] sshd-setup: Your identification has been saved in /etc/ssh/ssh_host_dsa_key.
[+] sshd-setup: Generating public/private dsa key pair.
[-] cgroup: + mount -t cgroup hugetlb -o hugetlb /sys/fs/cgroup/hugetlb/
[-] cgroup: + mkdir -p /sys/fs/cgroup/hugetlb
[-] cgroup: + mount -t cgroup net_prio -o net_prio /sys/fs/cgroup/net_prio/
[-] cgroup: + mkdir -p /sys/fs/cgroup/net_prio
[-] cgroup: + mount -t cgroup perf_event -o perf_event /sys/fs/cgroup/perf_event/
[-] cgroup: + mkdir -p /sys/fs/cgroup/perf_event
[-] cgroup: + mount -t cgroup net_cls -o net_cls /sys/fs/cgroup/net_cls/
[-] cgroup: + mkdir -p /sys/fs/cgroup/net_cls
[-] cgroup: + mount -t cgroup freezer -o freezer /sys/fs/cgroup/freezer/
[-] cgroup: + mkdir -p /sys/fs/cgroup/freezer
[-] cgroup: + mount -t cgroup devices -o devices /sys/fs/cgroup/devices/
[-] cgroup: + mkdir -p /sys/fs/cgroup/devices
[+] sshd-setup: +----[SHA256]-----+
[+] sshd-setup: |           o o.o.|
[+] sshd-setup: |          + . =.=|
[+] sshd-setup: |         o     Oo|
[+] sshd-setup: |         ..   o o|
[+] sshd-setup: |        S. E . o |
[+] sshd-setup: |       +.oo o + o|
[+] sshd-setup: |       .=oo+ o O |
[+] sshd-setup: |      o o*++  O  |
[+] sshd-setup: |     ..*++B. o.. |
[+] sshd-setup: +---[RSA 3072]----+
[+] sshd-setup: The key's randomart image is:
[+] sshd-setup: SHA256:k0w22eEhwIVdwF5rRGP5/M8FHBuUYw/8rv40bOOI8FA root@zero-os
[+] sshd-setup: The key fingerprint is:
[+] sshd-setup: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
[+] sshd-setup: Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
[+] sshd-setup: Generating public/private rsa key pair.
[-] cgroup: + mount -t cgroup memory -o memory /sys/fs/cgroup/memory/
[-] cgroup: + mkdir -p /sys/fs/cgroup/memory
[-] cgroup: + mount -t cgroup blkio -o blkio /sys/fs/cgroup/blkio/
[-] cgroup: + mkdir -p /sys/fs/cgroup/blkio
[-] cgroup: + mount -t cgroup cpuacct -o cpuacct /sys/fs/cgroup/cpuacct/
[-] cgroup: + mkdir -p /sys/fs/cgroup/cpuacct
[-] cgroup: + mount -t cgroup cpu -o cpu /sys/fs/cgroup/cpu/
[-] cgroup: + mkdir -p /sys/fs/cgroup/cpu
[-] cgroup: + mount -t cgroup cpuset -o cpuset /sys/fs/cgroup/cpuset/
[-] cgroup: + mkdir -p /sys/fs/cgroup/cpuset
[-] cgroup: + mount -t cgroup pids -o pids /sys/fs/cgroup/pids/
[+] sshd-setup: Setting up sshd
[-] cgroup: + mkdir -p /sys/fs/cgroup/pids
[-] cgroup: + subsys='pids cpuset cpu cpuacct blkio memory devices freezer net_cls perf_event net_prio hugetlb'
[-] haveged: haveged: listening socket at 3
[+] routing: Enable ip forwarding
[-] cgroup: + mount -t tmpfs cgroup_root /sys/fs/cgroup
@ashraffouda
Copy link
Collaborator

I assume it was stuck in zui, since node is functioning

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ashraffouda @scottyeager