unable to decode an event from the watch stream: context canceled

[gitmonster]

Hi, I need your help. Probably after an interrupted automatic HelmRepository update all flux-cd pods ran into CrashLoopBackOff.
I receive the following log in kustomize-controller. All other pods show an similar error. I tried to uninstall flux and then bootstrapping again but the issue persists. What can I do?

│{"level":"info","ts":"2024-07-13T15:41:01.713Z","logger":"setup","msg":"starting manager"}                                                                                                                                                                                                                                                                                │
│ {"level":"info","ts":"2024-07-13T15:41:01.713Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}                                                                                                                                                                                                                                                    │
│ {"level":"info","ts":"2024-07-13T15:41:01.713Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":":8080","secure":false}                                                                                                                                                                                                                │
│ {"level":"info","ts":"2024-07-13T15:41:01.714Z","msg":"starting server","name":"health probe","addr":":9440"}                                                                                                                                                                                                                                                             │
│ {"level":"info","ts":"2024-07-13T15:41:01.816Z","logger":"runtime","msg":"attempting to acquire leader lease flux-system/kustomize-controller-leader-election..."}                                                                                                                                                                                                        │
│ {"level":"info","ts":"2024-07-13T15:41:01.995Z","logger":"runtime","msg":"successfully acquired lease flux-system/kustomize-controller-leader-election"}                                                                                                                                                                                                                  │
│ {"level":"info","ts":"2024-07-13T15:41:01.997Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1.Kustomization"}                                                                                                                                     │
│ {"level":"info","ts":"2024-07-13T15:41:01.997Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1beta2.OCIRepository"}                                                                                                                                │
│ {"level":"info","ts":"2024-07-13T15:41:01.997Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1.GitRepository"}                                                                                                                                     │
│ {"level":"info","ts":"2024-07-13T15:41:01.997Z","msg":"Starting EventSource","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","source":"kind source: *v1beta2.Bucket"}                                                                                                                                       │
│ {"level":"info","ts":"2024-07-13T15:41:01.997Z","msg":"Starting Controller","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization"}                                                                                                                                                                                │
│ {"level":"info","ts":"2024-07-13T15:41:02.162Z","msg":"Starting workers","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","worker count":4}                                                                                                                                                                  │
│ {"level":"info","ts":"2024-07-13T15:41:03.404Z","msg":"request failed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"3003bf49-7cdc-4ecf-8555-7463001b9fa5","error":"Get \" │
│ http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/ad1a1682758d259038c1b8d8a82ea32e3d4cb7da.tar.gz\": dial tcp 10.43.25.84:80: connect: connection refused","method":"GET","url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/ad1a1682758d259038c1b8d8a82ea32e3d4cb7da.tar.g │
│ z"}                                                                                                                                                                                                                                                                                                                                                                       │
│ {"level":"info","ts":"2024-07-13T15:41:09.424Z","msg":"request failed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"3003bf49-7cdc-4ecf-8555-7463001b9fa5","error":"Get \" │
│ http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/ad1a1682758d259038c1b8d8a82ea32e3d4cb7da.tar.gz\": dial tcp 10.43.25.84:80: connect: connection refused","method":"GET","url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/ad1a1682758d259038c1b8d8a82ea32e3d4cb7da.tar.g │
│ z"}                                                                                                                                                                                                                                                                                                                                                                       │
│ {"level":"info","ts":"2024-07-13T15:41:20.464Z","msg":"request failed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"3003bf49-7cdc-4ecf-8555-7463001b9fa5","error":"Get \" │
│ http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/ad1a1682758d259038c1b8d8a82ea32e3d4cb7da.tar.gz\": dial tcp 10.43.25.84:80: connect: connection refused","method":"GET","url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/ad1a1682758d259038c1b8d8a82ea32e3d4cb7da.tar.g │
│ z"}                                                                                                                                                                                                                                                                                                                                                                       │
│ {"level":"info","ts":"2024-07-13T15:41:30.223Z","msg":"Stopping and waiting for non leader election runnables"}                                                                                                                                                                                                                                                           │
│ {"level":"info","ts":"2024-07-13T15:41:30.223Z","msg":"Stopping and waiting for leader election runnables"}                                                                                                                                                                                                                                                               │
│ {"level":"info","ts":"2024-07-13T15:41:30.223Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization"}                                                                                                                                        │
│ {"level":"info","ts":"2024-07-13T15:41:41.517Z","msg":"request failed","controller":"kustomization","controllerGroup":"kustomize.toolkit.fluxcd.io","controllerKind":"Kustomization","Kustomization":{"name":"flux-system","namespace":"flux-system"},"namespace":"flux-system","name":"flux-system","reconcileID":"3003bf49-7cdc-4ecf-8555-7463001b9fa5","error":"Get \" │
│ http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/ad1a1682758d259038c1b8d8a82ea32e3d4cb7da.tar.gz\": dial tcp 10.43.25.84:80: connect: connection refused","method":"GET","url":"http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/ad1a1682758d259038c1b8d8a82ea32e3d4cb7da.tar.g │
│ z"}                                                                                                                                                                                                                                                                                                                                                                       │
│ {"level":"info","ts":"2024-07-13T15:42:00.227Z","msg":"Stopping and waiting for caches"}                                                                                                                                                                                                                                                                                  │
│ {"level":"info","ts":"2024-07-13T15:42:00.227Z","msg":"Stopping and waiting for webhooks"}                                                                                                                                                                                                                                                                                │
│ {"level":"info","ts":"2024-07-13T15:42:00.227Z","msg":"Stopping and waiting for HTTP servers"}                                                                                                                                                                                                                                                                            │
│ {"level":"info","ts":"2024-07-13T15:42:00.227Z","msg":"Wait completed, proceeding to shutdown the manager"}                                                                                                                                                                                                                                                               │
│ {"level":"info","ts":"2024-07-13T15:42:00.227Z","logger":"runtime","msg":"k8s.io/[email protected]/tools/cache/reflector.go:232: watch of *v1.GitRepository ended with: an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding"}                                                        │
│ {"level":"info","ts":"2024-07-13T15:42:00.228Z","logger":"runtime","msg":"k8s.io/[email protected]/tools/cache/reflector.go:232: watch of *v1beta2.OCIRepository ended with: an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding"}                                                   │
│ {"level":"info","ts":"2024-07-13T15:42:00.228Z","logger":"runtime","msg":"k8s.io/[email protected]/tools/cache/reflector.go:232: watch of *v1beta2.Bucket ended with: an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding"}                                                          │
│ {"level":"info","ts":"2024-07-13T15:42:00.228Z","logger":"runtime","msg":"k8s.io/[email protected]/tools/cache/reflector.go:232: watch of *v1.Kustomization ended with: an error on the server (\"unable to decode an event from the watch stream: context canceled\") has prevented the request from succeeding"}                                                        │
│ {"level":"info","ts":"2024-07-13T15:42:00.228Z","msg":"shutting down server","name":"health probe","addr":":9440"}                                                                                                                                                                                                                                                        │
│ {"level":"info","ts":"2024-07-13T15:42:00.228Z","logger":"controller-runtime.metrics","msg":"Shutting down metrics server with timeout of 1 minute"}                                                                                                                                                                                                                      │
│ {"level":"error","ts":"2024-07-13T15:42:00.273Z","logger":"setup","msg":"problem running manager","error":"failed waiting for all runnables to end within grace period of 30s: context deadline exceeded"}                                                                                                                                                                │
│ Stream closed EOF for flux-system/kustomize-controller-5f58d55f76-ghsj5 (manager)                   

[gitmonster]

I found a solution to this network related problem and used --network-policy=false flag while bootstrapping. Not the safest way but works for now.