Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Virtual Kubelet - invalid memory address #2038

Closed
Sharathmk99 opened this issue Sep 21, 2023 · 2 comments · Fixed by #2039
Closed

Virtual Kubelet - invalid memory address #2038

Sharathmk99 opened this issue Sep 21, 2023 · 2 comments · Fixed by #2039

Comments

@Sharathmk99
Copy link
Contributor

What happened:

When upgrading from v0.9.1 to v0.9.4 manually upgraded virtualnodes.virtualkubelet.liqo.io CRD with right docker image and below args,

- --pod-reflection-workers=10
- --service-reflection-workers=3
- --endpointslice-reflection-workers=10
- --ingress-reflection-workers=0
- --configmap-reflection-workers=3
- --secret-reflection-workers=3
- --serviceaccount-reflection-workers=3
- --persistentvolumeclaim-reflection-workers=0
- --event-reflection-workers=3
- --service-reflection-type=AllowList
- --endpointslice-reflection-type=DenyList
- --ingress-reflection-type=AllowList
- --configmap-reflection-type=DenyList
- --secret-reflection-type=DenyList
- --event-reflection-type=DenyList
- --annotations-not-reflected=cloud.google.com/neg,cloud.google.com/neg-status,kubernetes.digitalocean.com/load-balancer-id,ingress.kubernetes.io/backends,ingress.kubernetes.io/forwarding-rule,ingress.kubernetes.io/target-proxy,ingress.kubernetes.io/url-map,metallb.universe.tf/address-pool,metallb.universe.tf/ip-allocated-from-pool,metallb.universe.tf/loadBalancerIPs

Updating VirtualNode CRD restarts the Virtual Kubelet Pod, but I saw below error and pod got restarted.

E0920 23:55:53.691200       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 146 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1edb2c0?, 0x39519d0})
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000d9b000?})
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b
panic({0x1edb2c0?, 0x39519d0?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/workload.(*NamespacedPodReflector).HandleStatus(0xc0005b6f00, {0x27eef60, 0xc000a76d20}, 0xc002ad2d80, 0xc00291db00, 0xc00200d4f0)
        /tmp/builder/pkg/virtualKubelet/reflection/workload/podns.go:412 +0xde7
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/workload.(*NamespacedPodReflector).Handle(0xc0005b6f00, {0x27eef60, 0xc000a76d20}, {0xc0042000f0, 0x2c})
        /tmp/builder/pkg/virtualKubelet/reflection/workload/podns.go:276 +0x1fa7
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/generic.(*reflector).handle(0xc0008e42d0, {0x27eef28, 0x39b66a0}, {{0xc002b70120?, 0x6b04520c28032005?}, {0xc0042000f0?, 0x7361654c65736165?}})
        /tmp/builder/pkg/virtualKubelet/reflection/generic/reflector.go:332 +0x9e2
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/generic.(*reflector).processNextWorkItem(0xc0008e42d0)
        /tmp/builder/pkg/virtualKubelet/reflection/generic/reflector.go:261 +0x3a7
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/generic.(*reflector).runWorker(...)
        /tmp/builder/pkg/virtualKubelet/reflection/generic/reflector.go:230
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xa341265736e6f70?, {0x27cd420, 0xc0009aa030}, 0x1, 0xc0007a8120)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x7265767265736463?, 0x3b9aca00, 0x0, 0x72?, 0x320031873726562?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x7265626d654d2e62?, 0x7265626d656d0752?, 0x6d654d130a252273?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161 +0x1e
created by github.com/liqotech/liqo/pkg/virtualKubelet/reflection/generic.(*reflector).Start in goroutine 1
        /tmp/builder/pkg/virtualKubelet/reflection/generic/reflector.go:154 +0x205
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1a4a567]

goroutine 146 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000d9b000?})
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xcd
panic({0x1edb2c0?, 0x39519d0?})
        /usr/local/go/src/runtime/panic.go:914 +0x21f
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/workload.(*NamespacedPodReflector).HandleStatus(0xc0005b6f00, {0x27eef60, 0xc000a76d20}, 0xc002ad2d80, 0xc00291db00, 0xc00200d4f0)
        /tmp/builder/pkg/virtualKubelet/reflection/workload/podns.go:412 +0xde7
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/workload.(*NamespacedPodReflector).Handle(0xc0005b6f00, {0x27eef60, 0xc000a76d20}, {0xc0042000f0, 0x2c})
        /tmp/builder/pkg/virtualKubelet/reflection/workload/podns.go:276 +0x1fa7
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/generic.(*reflector).handle(0xc0008e42d0, {0x27eef28, 0x39b66a0}, {{0xc002b70120?, 0x6b04520c28032005?}, {0xc0042000f0?, 0x7361654c65736165?}})
        /tmp/builder/pkg/virtualKubelet/reflection/generic/reflector.go:332 +0x9e2
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/generic.(*reflector).processNextWorkItem(0xc0008e42d0)
        /tmp/builder/pkg/virtualKubelet/reflection/generic/reflector.go:261 +0x3a7
github.com/liqotech/liqo/pkg/virtualKubelet/reflection/generic.(*reflector).runWorker(...)
        /tmp/builder/pkg/virtualKubelet/reflection/generic/reflector.go:230
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xa341265736e6f70?, {0x27cd420, 0xc0009aa030}, 0x1, 0xc0007a8120)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x7265767265736463?, 0x3b9aca00, 0x0, 0x72?, 0x320031873726562?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f
k8s.io/apimachinery/pkg/util/wait.Until(0x7265626d654d2e62?, 0x7265626d656d0752?, 0x6d654d130a252273?)
        /go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161 +0x1e
created by github.com/liqotech/liqo/pkg/virtualKubelet/reflection/generic.(*reflector).Start in goroutine 1
        /tmp/builder/pkg/virtualKubelet/reflection/generic/reflector.go:154 +0x205

After restart it started working fine. Not sure this is Crtical bug, just wanted to let you know.

What you expected to happen:

No Error

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Liqo version: v0.9.4
  • Liqoctl version: v0.9.4
  • Kubernetes version (use kubectl version): 1.22
  • Cloud provider or hardware configuration: kubeadm
  • Node image:
  • Network plugin and version:
  • Install tools:
  • Others:
@fra98 fra98 mentioned this issue Sep 21, 2023
Merged
@fra98
Copy link
Member

fra98 commented Sep 21, 2023

@Sharathmk99 thank you for sharing this issue. It will be fixed with #2039.

@Sharathmk99
Copy link
Contributor Author

Thank you!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix nil pointer deference in pod Ready condition fra98/liqo
2 participants
@Sharathmk99 @fra98