Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Race condition in kitctl bootstrap #386

Open
mengqiy opened this issue Mar 9, 2023 · 2 comments
Open

Race condition in kitctl bootstrap #386

mengqiy opened this issue Mar 9, 2023 · 2 comments
Labels
bug Something isn't working kitcli Label to group all tasks for kit cli

Comments

@mengqiy
Copy link
Contributor

mengqiy commented Mar 9, 2023 •
edited
Loading

There seems to be a circular dependency or race condition in kitctl bootstrap. There are dependencies listed below:

  • Management cluster control plane depends on kubelet.
  • Management cluster control plane needs to be ready to install CNI
  • kubelet will stop if CNI is not ready.

CP appeared to be ready at beginning and it become unavailable when some addons are being installed because the kubectl stopped due the CNI is not ready.

`kitctl bootstrap` log 
To access cluster run -

        export KUBECONFIG=/home/mengqiyu/.kit/env/mengqiyu-stormforge-9/etc/kubernetes/admin.conf

Installed chart https://aws.github.io/eks-charts/aws-vpc-cni
Applied https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.33.2/release.yaml
Applied https://storage.googleapis.com/tekton-releases/triggers/previous/v0.19.0/release.yaml
Applied https://github.com/tektoncd/dashboard/releases/download/v0.24.1/tekton-dashboard-release.yaml
reconciling addons.AWSLoadBalancer, err applying chart, installing chart: Get "https://52.26.212.203:8443/api/v1/namespaces/kube-system/services/aws-load-balancer-webhook-service": dial tcp 52.26.212.203:8443: connect: connection refused
reconciling addons.KITOperator, err applying chart, installing chart: Get "https://52.26.212.203:8443/api/v1/namespaces/kit/services/kit-webhook": dial tcp 52.26.212.203:8443: connect: connection refused                 
reconciling addons.Karpenter, err applying chart, installing chart: Get "https://52.26.212.203:8443/api/v1/namespaces/karpenter/services/karpenter": dial tcp 52.26.212.203:8443: connect: connection refused               
reconciling addons.AWSEBSCSIDriver, err applying chart, installing chart: Get "https://52.26.212.203:8443/apis/apps/v1/namespaces/kube-system/daemonsets/ebs-csi-node": dial tcp 52.26.212.203:8443: connect: connection refused
reconciling addons.PrometheusStack, err applying chart, installing chart: failed pre-install: timed out waiting for the condition
kubelet.service log 
Mar 09 02:31:23 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:23.751543    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:23 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:23.851974    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:23 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:23.952386    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:24.025669    2067 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.052428    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.152823    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.253337    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.353804    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.384019    2067 kubelet.go:2214] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.454078    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.554473    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.655376    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.755718    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:24 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:24.855988    2067 kubelet.go:2294] "Error getting node" err="node \"mengqiyu-stormforge-9\" not found"
Mar 09 02:31:25 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:25.052671    2067 apiserver.go:52] "Watching apiserver"
Mar 09 02:31:25 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:25.158484    2067 reconciler.go:157] "Reconciler: start to sync state"
Mar 09 02:31:29 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:29.026806    2067 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 09 02:31:29 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:29.388559    2067 kubelet.go:2214] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 09 02:31:34 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:34.027144    2067 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 09 02:31:34 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:34.394553    2067 kubelet.go:2214] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 09 02:31:39 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:39.027894    2067 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 09 02:31:39 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:39.400104    2067 kubelet.go:2214] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.377774    2067 topology_manager.go:187] "Topology Admit Handler"
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.377916    2067 topology_manager.go:187] "Topology Admit Handler"
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449735    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-proxy\" (UniqueName: \"kubernetes.io/configmap/decd34e8-ea2b-44cc-91fb-145421704e36-kube-proxy\") pod \"kube-proxy-drs69\" (UID: \"decd34e8-ea2b-44cc-91fb-145421704e36\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449780    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-bin-dir\" (UniqueName: \"kubernetes.io/host-path/55b51fac-adec-44d6-ad21-6ad9b9c24c9f-cni-bin-dir\") pod \"aws-node-crm6l\" (UID: \"55b51fac-adec-44d6-ad21-6ad9b9c24c9f\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449810    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/decd34e8-ea2b-44cc-91fb-145421704e36-xtables-lock\") pod \"kube-proxy-drs69\" (UID: \"decd34e8-ea2b-44cc-91fb-145421704e36\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449832    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dockershim\" (UniqueName: \"kubernetes.io/host-path/55b51fac-adec-44d6-ad21-6ad9b9c24c9f-dockershim\") pod \"aws-node-crm6l\" (UID: \"55b51fac-adec-44d6-ad21-6ad9b9c24c9f\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449854    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-dir\" (UniqueName: \"kubernetes.io/host-path/55b51fac-adec-44d6-ad21-6ad9b9c24c9f-log-dir\") pod \"aws-node-crm6l\" (UID: \"55b51fac-adec-44d6-ad21-6ad9b9c24c9f\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449876    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-dir\" (UniqueName: \"kubernetes.io/host-path/55b51fac-adec-44d6-ad21-6ad9b9c24c9f-run-dir\") pod \"aws-node-crm6l\" (UID: \"55b51fac-adec-44d6-ad21-6ad9b9c24c9f\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449903    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xtables-lock\" (UniqueName: \"kubernetes.io/host-path/55b51fac-adec-44d6-ad21-6ad9b9c24c9f-xtables-lock\") pod \"aws-node-crm6l\" (UID: \"55b51fac-adec-44d6-ad21-6ad9b9c24c9f\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449926    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/decd34e8-ea2b-44cc-91fb-145421704e36-lib-modules\") pod \"kube-proxy-drs69\" (UID: \"decd34e8-ea2b-44cc-91fb-145421704e36\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449953    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbvl\" (UniqueName: \"kubernetes.io/projected/55b51fac-adec-44d6-ad21-6ad9b9c24c9f-kube-api-access-bjbvl\") pod \"aws-node-crm6l\" (UID: \"55b51fac-adec-44d6-ad21-6ad9b9c24c9f\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.449982    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjx6j\" (UniqueName: \"kubernetes.io/projected/decd34e8-ea2b-44cc-91fb-145421704e36-kube-api-access-vjx6j\") pod \"kube-proxy-drs69\" (UID: \"decd34e8-ea2b-44cc-91fb-145421704e36\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.450005    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-net-dir\" (UniqueName: \"kubernetes.io/host-path/55b51fac-adec-44d6-ad21-6ad9b9c24c9f-cni-net-dir\") pod \"aws-node-crm6l\" (UID: \"55b51fac-adec-44d6-ad21-6ad9b9c24c9f\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.462824    2067 topology_manager.go:187] "Topology Admit Handler"
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:42.463057    2067 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="amazon-cloudwatch/fluent-bit-9w5p2" podUID=1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.551065    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlog\" (UniqueName: \"kubernetes.io/host-path/1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf-varlog\") pod \"fluent-bit-9w5p2\" (UID: \"1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.551112    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"fluent-bit-config\" (UniqueName: \"kubernetes.io/configmap/1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf-fluent-bit-config\") pod \"fluent-bit-9w5p2\" (UID: \"1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.551151    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dmesg\" (UniqueName: \"kubernetes.io/host-path/1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf-dmesg\") pod \"fluent-bit-9w5p2\" (UID: \"1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.551205    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlibdockercontainers\" (UniqueName: \"kubernetes.io/host-path/1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf-varlibdockercontainers\") pod \"fluent-bit-9w5p2\" (UID: \"1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.551230    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kb29\" (UniqueName: \"kubernetes.io/projected/1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf-kube-api-access-2kb29\") pod \"fluent-bit-9w5p2\" (UID: \"1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.551279    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"runlogjournal\" (UniqueName: \"kubernetes.io/host-path/1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf-runlogjournal\") pod \"fluent-bit-9w5p2\" (UID: \"1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf\") "
Mar 09 02:31:42 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:42.551373    2067 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"fluentbitstate\" (UniqueName: \"kubernetes.io/host-path/1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf-fluentbitstate\") pod \"fluent-bit-9w5p2\" (UID: \"1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf\") "
Mar 09 02:31:44 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:44.028907    2067 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 09 02:31:44 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:44.341799    2067 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="amazon-cloudwatch/fluent-bit-9w5p2" podUID=1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf
Mar 09 02:31:44 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:44.406755    2067 kubelet.go:2214] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 09 02:31:46 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:46.341951    2067 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="amazon-cloudwatch/fluent-bit-9w5p2" podUID=1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf
Mar 09 02:31:48 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:48.341879    2067 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="amazon-cloudwatch/fluent-bit-9w5p2" podUID=1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf
Mar 09 02:31:49 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:49.029201    2067 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 09 02:31:49 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:49.413083    2067 kubelet.go:2214] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 09 02:31:50 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:50.335890    2067 remote_runtime.go:334] "ContainerStatus from runtime service failed" err="rpc error: code = Unknown desc = Error: No such container: f31b6f4deb8a2f15e9a0f22fec5943ce3fce768124147b5bdb0baf0d3da89317" containerID="f31b6f4deb8a2f15e9a0f22fec5943ce3fce768124147b5bdb0baf0d3da89317"
Mar 09 02:31:50 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:50.335937    2067 kuberuntime_manager.go:1018] "getPodContainerStatuses for pod failed" err="rpc error: code = Unknown desc = Error: No such container: f31b6f4deb8a2f15e9a0f22fec5943ce3fce768124147b5bdb0baf0d3da89317" pod="kube-system/kube-proxy-drs69"
Mar 09 02:31:50 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:50.342120    2067 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="amazon-cloudwatch/fluent-bit-9w5p2" podUID=1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf
Mar 09 02:31:52 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:52.341782    2067 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="amazon-cloudwatch/fluent-bit-9w5p2" podUID=1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf
Mar 09 02:31:54 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: I0309 02:31:54.029890    2067 cni.go:239] "Unable to update cni config" err="no networks found in /etc/cni/net.d"
Mar 09 02:31:54 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:54.341881    2067 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="amazon-cloudwatch/fluent-bit-9w5p2" podUID=1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf
Mar 09 02:31:54 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:54.418371    2067 kubelet.go:2214] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
Mar 09 02:31:56 ip-10-4-238-247.us-west-2.compute.internal kubelet[2067]: E0309 02:31:56.341370    2067 pod_workers.go:190] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized" pod="amazon-cloudwatch/fluent-bit-9w5p2" podUID=1ccefa79-1ef2-4d33-9a4f-9e57559aa7bf
Mar 09 02:31:57 ip-10-4-238-247.us-west-2.compute.internal systemd[1]: Stopping kubelet.service...
Mar 09 02:31:57 ip-10-4-238-247.us-west-2.compute.internal systemd[1]: Stopped kubelet.service.

Manually restart the kubelet.service on the node will bring the management cluster to a working state.
@mengqiy mengqiy added bug Something isn't working kitcli Label to group all tasks for kit cli labels Mar 9, 2023
@mengqiy
Copy link
Contributor Author

mengqiy commented Mar 9, 2023

cc: @prateekgogia

@Ganiredi IIRC you have encounter similar issue

@Ganiredi
Copy link
Member

@mengqiy I encountered a similar issue where there was a circular dependency between the Management cluster control plane, kubelet, and the CNI plugin during the kitctl bootstrap process. The Management cluster control plane appeared to be ready initially but became unavailable when some addons were being installed due to the kubelet stopping because the CNI plugin was not ready.

@prateekgogia prateekgogia mentioned this issue Mar 17, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working kitcli Label to group all tasks for kit cli
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mengqiy @Ganiredi