Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Internal error occurred: failed calling webhook #334

Open
pau-hedgehog opened this issue Jan 21, 2025 · 1 comment
Open

Internal error occurred: failed calling webhook #334

pau-hedgehog opened this issue Jan 21, 2025 · 1 comment
Assignees
@pau-hedgehog

Comments

@pau-hedgehog
Copy link
Contributor

pau-hedgehog commented Jan 21, 2025 •
edited
Loading

Observed in env-3.

23:02:59 INF Configuring VPCs and VPCAttachments
23:03:06 INF sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: watch of *v1beta1.Connection ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
23:03:07 DBG Show tech collected successfully entry=server-6 output=/home/ubuntu/hhfab/show-tech-output/server-6-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-4 output=/home/ubuntu/hhfab/show-tech-output/server-4-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-3 output=/home/ubuntu/hhfab/show-tech-output/server-3-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-7 output=/home/ubuntu/hhfab/show-tech-output/server-7-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-8 output=/home/ubuntu/hhfab/show-tech-output/server-8-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-2 output=/home/ubuntu/hhfab/show-tech-output/server-2-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-10 output=/home/ubuntu/hhfab/show-tech-output/server-10-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-1 output=/home/ubuntu/hhfab/show-tech-output/server-1-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-9 output=/home/ubuntu/hhfab/show-tech-output/server-9-show-tech.log
23:03:07 DBG Show tech collected successfully entry=server-5 output=/home/ubuntu/hhfab/show-tech-output/server-5-show-tech.log
23:03:21 DBG Show tech collected successfully entry=as7326-01 output=/home/ubuntu/hhfab/show-tech-output/as7326-01-show-tech.log
23:03:24 DBG Show tech collected successfully entry=control-1 output=/home/ubuntu/hhfab/show-tech-output/control-1-show-tech.log
23:03:27 DBG Show tech collected successfully entry=as4630-01 output=/home/ubuntu/hhfab/show-tech-output/as4630-01-show-tech.log
23:03:28 DBG Show tech collected successfully entry=sse-c4632-02 output=/home/ubuntu/hhfab/show-tech-output/sse-c4632-02-show-tech.log
23:03:30 DBG Show tech collected successfully entry=s5248-05 output=/home/ubuntu/hhfab/show-tech-output/s5248-05-show-tech.log
23:03:38 DBG Show tech collected successfully entry=as7712-01 output=/home/ubuntu/hhfab/show-tech-output/as7712-01-show-tech.log
23:03:41 DBG Show tech collected successfully entry=as7712-02 output=/home/ubuntu/hhfab/show-tech-output/as7712-02-show-tech.log
23:03:41 INF Show tech files saved in folder=/home/ubuntu/hhfab/show-tech-output
23:03:41 WRN Error running on-ready commands err="setting up VPCs: deleting attachment \"server-6--unbundled--as4630-01--vpc-06--subnet-01\": Internal error occurred: failed calling webhook \"vvpcattachment.kb.io\": failed to call webhook: Post \"https://fabric-webhook-service.fab.svc:443/validate-vpc-githedgehog-com-v1beta1-vpcattachment?timeout=10s\": proxy error from 127.0.0.1:6443 while dialing 172.28.0.47:9443, code 502: 502 Bad Gateway"
23:03:41 WRN Failed running VM vm=server-7 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-5 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-2 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-9 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-8 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-3 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-10 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-4 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-6 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=server-1 type=server err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 WRN Failed running VM vm=control-1 type=control err="running \"qemu-system-x86_64\": signal: killed"
23:03:41 ERR running VLAB: running task: running on-ready commands: setting up VPCs: deleting attachment "server-6--unbundled--as4630-01--vpc-06--subnet-01": Internal error occurred: failed calling webhook "vvpcattachment.kb.io": failed to call webhook: Post "https://fabric-webhook-service.fab.svc:443/validate-vpc-githedgehog-com-v1beta1-vpcattachment?timeout=10s": proxy error from 127.0.0.1:6443 while dialing 172.28.0.47:9443, code 502: 502 Bad Gateway

I saw fabricator-ctrl in CrashLoopBackOff:

NAMESPACE     NAME                                       READY   STATUS             RESTARTS      AGE   IP            NODE        NOMINATED NODE   READINESS GATES
fab           cert-manager-755d45f55-kvjrg               1/1     Running            2 (49s ago)   57m   172.28.0.61   control-1   <none>           <none>
fab           cert-manager-cainjector-65bb576cc5-k7qnv   1/1     Running            2 (49s ago)   57m   172.28.0.55   control-1   <none>           <none>
fab           cert-manager-webhook-8664b448c6-zkslp      0/1     Running            2 (49s ago)   57m   172.28.0.53   control-1   <none>           <none>
fab           fabric-boot-54bb58c8cd-n9lxs               0/1     Running            4 (49s ago)   56m   172.30.0.5    control-1   <none>           <none>
fab           fabric-ctrl-769f4cc57c-s22n7               0/1     Running            2 (49s ago)   56m   172.28.0.56   control-1   <none>           <none>
fab           fabric-dhcpd-6d4954695c-lb6l9              1/1     Running            2 (49s ago)   57m   172.30.0.5    control-1   <none>           <none>
fab           fabric-proxy-7bdf694dc9-wfx52              1/1     Running            2 (49s ago)   57m   172.28.0.52   control-1   <none>           <none>
fab           fabricator-ctrl-848fcb8dd9-zrssd           0/1     CrashLoopBackOff   2 (4s ago)    56m   172.28.0.57   control-1   <none>           <none>
fab           helm-install-cert-manager-752f8            0/1     Completed          2             57m   <none>        control-1   <none>           <none>
fab           helm-install-fabric-2ftvl                  0/1     Completed          2             57m   <none>        control-1   <none>           <none>
fab           helm-install-fabric-api-mtzvk              0/1     Completed          2             57m   <none>        control-1   <none>           <none>
fab           helm-install-fabric-boot-klzq9             0/1     Completed          2             57m   <none>        control-1   <none>           <none>
fab           helm-install-fabric-dhcpd-cct6t            0/1     Completed          2             57m   <none>        control-1   <none>           <none>
fab           helm-install-fabric-proxy-bvqrm            0/1     Completed          2             57m   <none>        control-1   <none>           <none>
fab           helm-install-fabricator-api-8dg52          0/1     Completed          0             58m   <none>        control-1   <none>           <none>
fab           helm-install-fabricator-wskxg              0/1     Completed          0             58m   <none>        control-1   <none>           <none>
fab           helm-install-ntp-9n8jf                     0/1     Completed          0             56m   <none>        control-1   <none>           <none>
fab           helm-install-reloader-xmv9k                0/1     Completed          0             57m   <none>        control-1   <none>           <none>
fab           helm-install-zot-hp6lc                     0/1     Completed          0             57m   <none>        control-1   <none>           <none>
fab           ntp-7b5fdc4f86-9xms8                       0/1     Running            2 (49s ago)   56m   172.30.0.5    control-1   <none>           <none>
fab           reloader-reloader-5f6cc54f89-mv2df         0/1     Running            2 (49s ago)   57m   172.28.0.54   control-1   <none>           <none>
fab           zot-d84f4b65d-j44vx                        0/1     Running            2 (49s ago)   56m   172.28.0.62   control-1   <none>           <none>
kube-system   coredns-56f6fc8fd7-g7rp2                   1/1     Running            2 (49s ago)   60m   172.28.0.58   control-1   <none>           <none>
kube-system   local-path-provisioner-846b9dcb6c-q24wh    1/1     Running            2 (49s ago)   60m   172.28.0.60   control-1   <none>           <none>
kube-system   metrics-server-5985cbc9d7-pwlhb            0/1     Running            2 (49s ago)   60m   172.28.0.59   control-1   <none>           <none>

Unfortunately not enough logs could be collected and inspected
@pau-hedgehog pau-hedgehog self-assigned this Jan 21, 2025
@pau-hedgehog pau-hedgehog changed the title Internal error occurred: failed calling webhook "vvpcattachment.kb.io Internal error occurred: failed calling webhook Jan 21, 2025
@Frostman
Copy link
Member

It looks like API server was down at that time :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pau-hedgehog pau-hedgehog

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Frostman @pau-hedgehog