kubelet service switch between NodeNotSchedulable and NodeSchedulable frequently

[antzjm]

[provide a description of the issue]

Version

4.8.9

Steps To Reproduce

Current Result

The kubelet service change from NodeSchedulable to NodeNotSchedulable and turn back frequently.
Events:
Type Reason Age From Message

---

Normal NodeNotSchedulable 93m (x30 over 10h) kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeNotSchedulable
Normal NodeSchedulable 91m (x30 over 10h) kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeSchedulable
Normal Starting 81m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 81m kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 81m kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 81m kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeHasSufficientPID
Normal NodeNotReady 81m kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeNotReady
Normal NodeAllocatableEnforced 81m kubelet Updated Node Allocatable limit across pods
Normal NodeReady 81m kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeReady
Normal NodeNotSchedulable 3m19s (x5 over 75m) kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeNotSchedulable
Normal NodeSchedulable 118s (x5 over 73m) kubelet Node hztt-oe19-05-snc-2-18723 status is now: NodeSchedulable

Expected Result

I hope find the root reason of this issue. Does OCP add some monitor to change node schedulable all day long?

Additional Information

[if you are reporting issue related to builds, provide build logs with BUILD_LOGLEVEL=5]
[consider attaching output of the $ oc get all -o json -n <namespace> command to the issue]
[visit https://docs.openshift.org/latest/welcome/index.html]

[vrutkovs]

Check .metadata.managedFields to see which component has changed schedulability

[antzjm]

Check which resource, node? or pod?

[antzjm]

There is no managedFields field in node resource.

[antzjm]

Here is error log of kubelet:
Pool master: node hztt-oe19-05-snc-2-18723: Reporting unready: node hztt-oe19-05-snc-2-18723 is reporting Unschedulable