panic: kube controller runtime cache: failed to watch: the server is currently unable to handle the request

[pau-hedgehog]

Seen in: https://github.com/githedgehog/fabricator/actions/runs/12874768153/job/35895431191

21:44:04 ERR kube controller runtime cache: failed to watch err="the server is currently unable to handle the request (get switches.wiring.githedgehog.com) - error from a previous attempt: EOF"
21:44:04 ERR kube controller runtime cache: failed to watch err="the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: EOF"
21:44:04 ERR kube controller runtime cache: failed to watch err="the server is currently unable to handle the request (get servers.wiring.githedgehog.com) - error from a previous attempt: EOF"
21:44:04 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.Switch: the server is currently unable to handle the request (get switches.wiring.githedgehog.com) - error from a previous attempt: EOF"
21:44:04 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.IPv4Namespace: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: EOF"
21:44:04 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.Server: the server is currently unable to handle the request (get servers.wiring.githedgehog.com) - error from a previous attempt: EOF"
21:44:04 ERR kube controller runtime cache: failed to watch err="the server is currently unable to handle the request (get connections.wiring.githedgehog.com) - error from a previous attempt: EOF"
21:44:04 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.Connection: the server is currently unable to handle the request (get connections.wiring.githedgehog.com) - error from a previous attempt: EOF"
21:44:04 ERR Observed a panic panic="kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get connections.wiring.githedgehog.com) - error from a previous attempt: EOF" panicGoValue="&fmt.wrapError{msg:\"kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get connections.wiring.githedgehog.com) - error from a previous attempt: EOF\", err:(*rest.wrapPreviousError)(0xc0005c15a0)}" stacktrace="goroutine 136133 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc0030537c0})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc\nk8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc0030537c0}, {0x8ae06c0, 0x0, 0x43bf25?})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc005784380?})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108\npanic({0x2e10700?, 0xc0030537c0?})\n\t/home/runner/_work/_tool/go/1.23.4/x64/src/runtime/panic.go:785 +0x132\ngo.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc000eac4b0, {0x39c7dc0, 0xc0005c15a0})\n\t/home/runner/_work/fabricator/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00108df50, {0x39c7700, 0xc0022463c0}, 0x1, 0xc001ef68c0)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/client-go/tools/cache.(*Reflector).Run(0xc000eac4b0, 0xc001ef68c0)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c\ncreated by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136104\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73\n"
21:44:04 ERR Observed a panic panic="kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get servers.wiring.githedgehog.com) - error from a previous attempt: EOF" panicGoValue="&fmt.wrapError{msg:\"kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get servers.wiring.githedgehog.com) - error from a previous attempt: EOF\", err:(*rest.wrapPreviousError)(0xc001eb1c40)}" stacktrace="goroutine 136058 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc00286a6a0})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc\nk8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc00286a6a0}, {0x8ae06c0, 0x0, 0x43bf25?})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc002ffefc0?})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108\npanic({0x2e10700?, 0xc00286a6a0?})\n\t/home/runner/_work/_tool/go/1.23.4/x64/src/runtime/panic.go:785 +0x132\ngo.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc0009de000, {0x39c7dc0, 0xc001eb1c40})\n\t/home/runner/_work/fabricator/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000f85f50, {0x39c7700, 0xc00305c140}, 0x1, 0xc001a1a850)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/client-go/tools/cache.(*Reflector).Run(0xc0009de000, 0xc001a1a850)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c\ncreated by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136092\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73\n"
21:44:04 ERR Observed a panic panic="kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get switches.wiring.githedgehog.com) - error from a previous attempt: EOF" panicGoValue="&fmt.wrapError{msg:\"kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get switches.wiring.githedgehog.com) - error from a previous attempt: EOF\", err:(*rest.wrapPreviousError)(0xc004b024e0)}" stacktrace="goroutine 136140 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc001eb1d20})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc\nk8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc001eb1d20}, {0x8ae06c0, 0x0, 0x43bf25?})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0022828c0?})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108\npanic({0x2e10700?, 0xc001eb1d20?})\n\t/home/runner/_work/_tool/go/1.23.4/x64/src/runtime/panic.go:785 +0x132\ngo.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc000eac960, {0x39c7dc0, 0xc004b024e0})\n\t/home/runner/_work/fabricator/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000f83f50, {0x39c7700, 0xc002246500}, 0x1, 0xc001ef6d20)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/client-go/tools/cache.(*Reflector).Run(0xc000eac960, 0xc001ef6d20)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c\ncreated by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136107\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73\n"
21:44:04 ERR Observed a panic panic="kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: EOF" panicGoValue="&fmt.wrapError{msg:\"kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: EOF\", err:(*rest.wrapPreviousError)(0xc001eb1be0)}" stacktrace="goroutine 135100 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc004ddd2a0})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc\nk8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc004ddd2a0}, {0x8ae06c0, 0x0, 0x43bf25?})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001fb9340?})\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108\npanic({0x2e10700?, 0xc004ddd2a0?})\n\t/home/runner/_work/_tool/go/1.23.4/x64/src/runtime/panic.go:785 +0x132\ngo.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc000eac000, {0x39c7dc0, 0xc001eb1be0})\n\t/home/runner/_work/fabricator/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00120ff50, {0x39c7700, 0xc0022460f0}, 0x1, 0xc004db98f0)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/client-go/tools/cache.(*Reflector).Run(0xc000eac000, 0xc004db98f0)\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c\ncreated by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136101\n\t/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73\n"
panic: kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get connections.wiring.githedgehog.com) - error from a previous attempt: EOF [recovered]
	panic: kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get connections.wiring.githedgehog.com) - error from a previous attempt: EOF

goroutine 136133 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc0030537c0}, {0x8ae06c0, 0x0, 0x43bf25?})
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:89 +0xee
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc005784380?})
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
panic({0x2e10700?, 0xc0030537c0?})
	/home/runner/_work/_tool/go/1.23.4/x64/src/runtime/panic.go:785 +0x132
go.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc000eac4b0, {0x39c7dc0, 0xc0005c15a0})
	/home/runner/_work/fabricator/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00108df50, {0x39c7700, 0xc0022463c0}, 0x1, 0xc001ef68c0)
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000eac4b0, 0xc001ef68c0)
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 136104
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73
panic: kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: EOF [recovered]
	panic: kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: EOF

goroutine 135100 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a020d8, 0x8ae06c0}, {0x2e10700, 0xc004ddd2a0}, {0x8ae06c0, 0x0, 0x43bf25?})
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:89 +0xee
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc001fb9340?})
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
panic({0x2e10700?, 0xc004ddd2a0?})
	/home/runner/_work/_tool/go/1.23.4/x64/src/runtime/panic.go:785 +0x132
go.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc000eac000, {0x39c7dc0, 0xc001eb1be0})
	/home/runner/_work/fabricator/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00120ff50, {0x39c7700, 0xc0022460f0}, 0x1, 0xc004db98f0)
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc000eac000, 0xc004db98f0)
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 13[610](https://github.com/githedgehog/fabricator/actions/runs/12874768153/job/35895431191#step:7:611)1
	/home/runner/_work/fabricator/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

[pau-hedgehog]

Also seen in env-3:

18:40:04 INF Switches status ready=[] notReady="[as4630-01 as7326-01 as7712-01 as7712-02 s5248-05 sse-c4632-02]"
18:40:18 ERR kube controller runtime cache: failed to watch err="context deadline exceeded"
18:40:18 ERR kube controller runtime cache: failed to watch err="context deadline exceeded"
18:40:18 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.Server: context deadline exceeded"
18:40:18 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.VLANNamespace: context deadline exceeded"
18:40:18 ERR Observed a panic panic="kube controller runtime cache: failed to watch: context deadline exceeded" panicGoValue="&fmt.wrapError{msg:\"kube controller runtime cache: failed to watch: context deadline exceeded\", err:context.deadlineExceededError{}}" stacktrace="goroutine 1063 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3a3f4d8, 0x8a297e0}, {0x2e44340, 0xc004c800e0})\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc\nk8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a3f4d8, 0x8a297e0}, {0x2e44340, 0xc004c800e0}, {0x8a297e0, 0x0, 0x43bf25?})\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc006585dc0?})\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108\npanic({0x2e44340?, 0xc004c800e0?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngo.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc0010b2000, {0x3a03060, 0x8a297e0})\n\t/home/ubuntu/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1()\n\t/home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc005ee7f50, {0x3a048c0, 0xc006a2a140}, 0x1, 0xc006d155e0)\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/client-go/tools/cache.(*Reflector).Run(0xc0010b2000, 0xc006d155e0)\n\t/home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c\ncreated by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 1057\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73\n"
18:40:18 ERR Observed a panic panic="kube controller runtime cache: failed to watch: context deadline exceeded" panicGoValue="&fmt.wrapError{msg:\"kube controller runtime cache: failed to watch: context deadline exceeded\", err:context.deadlineExceededError{}}" stacktrace="goroutine 1046 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x3a3f4d8, 0x8a297e0}, {0x2e44340, 0xc005c7c600})\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc\nk8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a3f4d8, 0x8a297e0}, {0x2e44340, 0xc005c7c600}, {0x8a297e0, 0x0, 0x43bf25?})\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc006604e00?})\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108\npanic({0x2e44340?, 0xc005c7c600?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngo.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc0008ac1e0, {0x3a03060, 0x8a297e0})\n\t/home/ubuntu/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1()\n\t/home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001e13f50, {0x3a048c0, 0xc006570140}, 0x1, 0xc006dbd180)\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/client-go/tools/cache.(*Reflector).Run(0xc0008ac1e0, 0xc006dbd180)\n\t/home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c\ncreated by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 448\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73\n"
panic: kube controller runtime cache: failed to watch: context deadline exceeded [recovered]
        panic: kube controller runtime cache: failed to watch: context deadline exceeded

goroutine 1063 [running]:
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a3f4d8, 0x8a297e0}, {0x2e44340, 0xc004c800e0}, {0x8a297e0, 0x0, 0x43bf25?})
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:89 +0xee
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc006585dc0?})
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108
panic({0x2e44340?, 0xc004c800e0?})
        /usr/local/go/src/runtime/panic.go:785 +0x132
go.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc0010b2000, {0x3a03060, 0x8a297e0})
        /home/ubuntu/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()
        /home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc005ee7f50, {0x3a048c0, 0xc006a2a140}, 0x1, 0xc006d155e0)
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc0010b2000, 0xc006d155e0)
        /home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 1057
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

[pau-hedgehog]

And today on my VLAB, after restarting it:

ubuntu@vlab-pau:~/hhfab$ ./hhfab-reinstall vlab up -v                                                                                                                                                                              
16:19:17 INF Hedgehog Fabricator version=v0.34.1-9-g92f5d34e-E20926-pc 
...
16:20:15 INF fabric-install: Feb 05 16:20:10 control-1 systemd[1]: fabric-install.service - "Firstboot installation program for Hedgehog Fabricator" was skipped because of an unmet condition check (ConditionPathExists=!/opt/hedge
hog/.install). vm=control-1                                                                                                                                                                                                          
16:20:20 INF Control node is ready vm=control-1 type=control                                                                                                                                                                         
16:20:20 DBG VM is ready vm=control-1 type=control                                                                                                                                                                                   
16:20:20 INF Journalctl on control node failed vm=control-1 type=control err="running ssh command \"journalctl -n 100 -fu fabric-install.service\": context canceled"  

Configuring VPCs:

16:21:31 INF Hedgehog Fabricator version=v0.34.1-6-g938bd3ad-dirty-9b1611-pc                                                                                                                                                         
16:21:31 INF Wiring hydrated successfully mode=if-not-present                                                                                                                                                                        
16:21:31 INF VLAB config loaded file=vlab/config.yaml                                                                                                                                                                                
16:21:31 INF Setting up VPCs and VPCAttachments perSubnet=1 perVPC=1 wait=true cleanup=true                                                                                                                                          
16:21:33 INF Waiting for switches ready before configuring VPCs and VPCAttachments                                                                                                                                                   
16:21:33 INF Switches status ready=[] notReady="[leaf-01 leaf-02 leaf-03 leaf-04 leaf-05 spine-01 spine-02]"                                                                                                                         
...
16:25:27 INF Switches status ready=[] notReady="[leaf-01 leaf-02 leaf-03 leaf-04 leaf-05 spine-01 spine-02]"                                                                                                                         16:25:42 ERR kube controller runtime cache: failed to watch err="the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52536->127.0.0.1:6
443: read: connection reset by peer"                                                                                                                                                                                                 16:25:42 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.IPv4Namespace: the server is currently unable to handle the request (get ipv4nam
espaces.vpc.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52536->127.0.0.1:6443: read: connection reset by peer"                                                                                              
16:25:42 ERR kube controller runtime cache: failed to watch err="the server is currently unable to handle the request (get connections.wiring.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52526->127.0.0.1:6443: read: connection reset by peer"                                                                                                                                                                                                 
16:25:42 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.Connection: the server is currently unable to handle the request (get connection
s.wiring.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52526->127.0.0.1:6443: read: connection reset by peer"                                                                                                 16:25:42 ERR kube controller runtime cache: failed to watch err="the server is currently unable to handle the request (get switches.wiring.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52548->127.0.0.1:6443
: read: connection reset by peer"                                                                                                                                                                                                    
16:25:42 ERR Unhandled Error logger=UnhandledError err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1beta1.Switch: the server is currently unable to handle the request (get switches.wirin
g.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52548->127.0.0.1:6443: read: connection reset by peer"                                                                                                        16:25:42 ERR Observed a panic panic="kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.
1:52536->127.0.0.1:6443: read: connection reset by peer" panicGoValue="&fmt.wrapError{msg:\"kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52536->127.0.0.1:6443: read: connection reset by peer\", err:(*rest.wrapPreviousError)(0xc0042c07e0)}" stacktrace="goroutine 119 [running]:\nk8s.io/apimachinery/pkg/util/ru
ntime.logPanic({0x3a64b78, 0x8a60c40}, {0x2e622e0, 0xc0042c0880})\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:107 +0xbc\nk8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a64b78, 0x8a60c40
}, {0x2e622e0, 0xc0042c0880}, {0x8a60c40, 0x0, 0x43bf25?})\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:82 +0x5e\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc004ef08c0?})\n\
t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108\npanic({0x2e622e0?, 0xc0042c0880?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngo.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc004ef2000, {0x3a2a5e0, 0xc0042c07e0})\n\t/home/ubuntu/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa\nk8s.io/client-go/tools/cache.(*Reflector).Run.func1()\n\t/home/ubuntu/fabric
ator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc005087f50, {0x3a29f20, 0xc004ee8140}, 0x1, 0xc004eee150)\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/client-go/tools/cache.(*Reflector).Run(
0xc004ef2000, 0xc004eee150)\n\t/home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af\nk8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c\ncreated by k8s.io/apimachinery/p
kg/util/wait.(*Group).Start in goroutine 113\n\t/home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73\n"                                                                                                 
panic: kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52536->127.0.0.1:6443: read:
 connection reset by peer [recovered]                                                                                                                                                                                                
        panic: kube controller runtime cache: failed to watch: the server is currently unable to handle the request (get ipv4namespaces.vpc.githedgehog.com) - error from a previous attempt: read tcp 127.0.0.1:52536->127.0.0.1:644
3: read: connection reset by peer                                                                                                                                                                                                    
                                                                                                                                                                                                                                     
goroutine 119 [running]:                                                                                                                                                                                                             
k8s.io/apimachinery/pkg/util/runtime.handleCrash({0x3a64b78, 0x8a60c40}, {0x2e622e0, 0xc0042c0880}, {0x8a60c40, 0x0, 0x43bf25?})                                                                                                     
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:89 +0xee                                                                                                                                      
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc004ef08c0?})                                       
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:59 +0x108                                                                                                                                     
panic({0x2e622e0?, 0xc0042c0880?})              
        /usr/local/go/src/runtime/panic.go:785 +0x132                                                             
go.githedgehog.com/fabric/pkg/util/kubeutil.cacheWatchErrorHandler(0xc004ef2000, {0x3a2a5e0, 0xc0042c07e0})
        /home/ubuntu/fabricator/vendor/go.githedgehog.com/fabric/pkg/util/kubeutil/kube.go:128 +0x1fa             
k8s.io/client-go/tools/cache.(*Reflector).Run.func1()                                                             
        /home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:308 +0x44
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x10?)                                                 
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:226 +0x33               
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc005087f50, {0x3a29f20, 0xc004ee8140}, 0x1, 0xc004eee150)  
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:227 +0xaf               
k8s.io/client-go/tools/cache.(*Reflector).Run(0xc004ef2000, 0xc004eee150)                                   
        /home/ubuntu/fabricator/vendor/k8s.io/client-go/tools/cache/reflector.go:306 +0x1af                 
k8s.io/client-go/tools/cache.(*controller).Run.(*Group).StartWithChannel.func2()                            
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:55 +0x1b                   
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()                                                          
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72 +0x4c                   
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start in goroutine 113                                
        /home/ubuntu/fabricator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:70 +0x73

Checking the control VM:

core@control-1 ~ $ kubectl -n fab get pods
NAME                                       READY   STATUS      RESTARTS       AGE
cert-manager-755d45f55-xpxdd               1/1     Running     11 (49m ago)   45h
cert-manager-cainjector-65bb576cc5-hq5jj   1/1     Running     9 (49m ago)    45h
cert-manager-webhook-8664b448c6-pxjdt      1/1     Running     8 (55m ago)    45h
fabric-boot-55f986598b-xrms9               1/1     Running     10 (55m ago)   45h
fabric-ctrl-79b564d89d-gchj5               1/1     Running     14 (49m ago)   45h
fabric-dhcpd-5875dfd968-cxrtl              1/1     Running     13 (49m ago)   45h
fabric-proxy-cfd89bbb-pqmfp                1/1     Running     8 (55m ago)    45h
fabricator-ctrl-7bf88dd97d-n2k9t           1/1     Running     18 (47m ago)   45h
helm-install-cert-manager-8vhk5            0/1     Completed   0              45h
helm-install-fabric-api-5kkxz              0/1     Completed   0              45h
helm-install-fabric-boot-cbm5s             0/1     Completed   0              45h
helm-install-fabric-dhcpd-6dbj4            0/1     Completed   0              45h
helm-install-fabric-ghg4g                  0/1     Completed   0              45h
helm-install-fabric-proxy-5qnjg            0/1     Completed   0              45h
helm-install-fabricator-api-mf28w          0/1     Completed   0              45h
helm-install-fabricator-ljfpg              0/1     Completed   0              45h
helm-install-ntp-85c4k                     0/1     Completed   0              45h
helm-install-reloader-md2mv                0/1     Completed   0              45h
helm-install-zot-4fj79                     0/1     Completed   0              45h
ntp-7b5fdc4f86-9dm7w                       1/1     Running     8 (55m ago)    45h
reloader-reloader-5f6cc54f89-xqxgm         1/1     Running     8 (55m ago)    45h
zot-66f75ff8b8-p5d82                       1/1     Running     11 (55m ago)   45h

control-1-show-tech.log

[Frostman]

it seems like on a vlab restart we aren't giving control node enough time to boot up, I'll have a look

[pau-hedgehog]

Possible it on the VLAB CI:
https://github.com/githedgehog/fabricator/actions/runs/13161517426/job/36731821026?pr=371

[pau-hedgehog]

A pair of hits today:
https://github.com/githedgehog/fabricator/actions/runs/13187154996/job/36831984994
https://github.com/githedgehog/fabricator/actions/runs/13187154996/job/36831984308