upgrading kernel version of master node cause apiserver keeps restarting

[pulsarrrr]

My environment：
kubernetes：1.24.17 (3 masters and 1 worker)
etcd and kube-apiserver run as pod in my cluster.

It runs well when all of 3 masters are 3.10.0-1160.99.1.el7.x86_64 kernel version;

I upgraded one master's kernel version to 5.4.277-1.el7.elrepo.x86_64(delete this node—>upgrade kernel version—>join the cluster)

this master's kube-apiserver runs well within the first hour, then it begins restarting and make etcd restarting, Besides, kube-apiserver makes the node stuck, after mv kube-apiserver.yaml from /etc/kubernets/manifests to somewhere else ,my node becomes normal.

I can't figure out what happened to my apiserver, any help would be appreciate.

The following is the related logs of kube-apiserver.

E0904 07:37:58.681591       1 available_controller.go:524] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.1.19.38:443/apis/metrics.k8s.io/v1beta1: Get "https://10.1.19.38:443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

E0904 07:38:01.593119       1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
I0904 07:38:01.593177       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0904 07:38:01.913053       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: error trying to reach service: dial tcp 10.1.19.38:443: connect: connection timed out

E0904 07:38:36.200130       1 available_controller.go:524] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again

E0904 08:43:43.765735       1 status.go:71] apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded
E0904 08:43:43.765855       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0904 08:43:43.767076       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0904 08:43:43.768363       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0904 08:43:43.769740       1 trace.go:205] Trace[1304417108]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.24.17 (linux/amd64) kubernetes/22a9682/leader-election,audit-id:d139ac48-6f60-4895-b634-5260be768ba8,client:10.1.69.88,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (04-Sep-2024 08:43:38.780) (total time: 4989ms):
Trace[1304417108]: [4.989288367s] [4.989288367s] END
E0904 08:43:43.779268       1 timeout.go:141] post-timeout activity - time-elapsed: 13.468579ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager" result: <nil>
{"level":"warn","ts":"2024-09-04T08:43:43.965Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a15340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
{"level":"warn","ts":"2024-09-04T08:43:45.338Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc002013880/127.0.0.1:2379","attempt":0,"error":"rpc error: code = Canceled desc = context canceled"}
E0904 08:43:45.338314       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
E0904 08:43:45.338398       1 writers.go:118] apiserver was unable to write a JSON response: http: Handler timeout
E0904 08:43:45.339620       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0904 08:43:45.340857       1 writers.go:131] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0904 08:43:45.342200       1 trace.go:205] Trace[1213158620]: "Get" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:kube-scheduler/v1.24.17 (linux/amd64) kubernetes/22a9682/leader-election,audit-id:e034f986-8083-4c20-b66c-029720afa2ba,client:10.1.69.88,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (04-Sep-2024 08:43:40.347) (total time: 4994ms):
Trace[1213158620]: [4.994932702s] [4.994932702s] END
E0904 08:43:45.347125       1 timeout.go:141] post-timeout activity - time-elapsed: 8.904939ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: <nil>

E0904 08:45:01.625216       1 controller.go:220] unable to create required kubernetes system namespace kube-public: Internal error occurred: resource quota evaluation timed out
E0904 08:45:01.626394       1 controller.go:220] unable to create required kubernetes system namespace kube-node-lease: Post "https://[::1]:6443/api/v1/namespaces": dial tcp [::1]:6443: connect: connection refused
{"level":"warn","ts":"2024-09-04T08:45:01.679Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000a15340/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\""}

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[dims]

a better place to ask would be on the support channels. please see:
https://git.k8s.io/kubernetes/SUPPORT.md

/kind support
/close

[k8s-ci-robot]

@dims: Closing this issue.

In response to this:

a better place to ask would be on the support channels. please see:
https://git.k8s.io/kubernetes/SUPPORT.md

/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.