KCP: Consider reconciling on Pod events

[sbueringer]

We recently changed KCP to use a cached client to retrieve the Pods we care about.

This means we already have informers for Pods of workload clusters.

I'm wondering if we should also trigger reconciles if these Pods change (similar to how we watch on Node events in the Machine controller).

Pro

• we can instantly react to Pod changes instead of using requeue after

Con

• we get probably more reconciles
  • This can be mitigated by only letting the update events through that we care about or by rate-limiting. This might even lead to less reconciles when we get rid of the requeues we have because we don't watch on Pod events (

    cluster-api/controlplane/kubeadm/internal/controllers/controller.go

    Line 253 in d934125

    res = ctrl.Result{RequeueAfter: 20 * time.Second}

    )

[chrischdi]

/triage accepted

[chrischdi]

/help

This requires some research about how to avoid too many reconciles + scale tests.

[k8s-ci-robot]

@chrischdi:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

This requires some research about how to avoid too many reconciles + scale tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[arshadd-b]

/assign

[arshadd-b]

Hi @sbueringer
I have found where we are watching on node events in machine controller here

But this change I could not find in KCP We recently changed KCP to use a cached client to retrieve the Pods we care about.
Could you please help here?
Thanks

[chrischdi]

The change in KCP was this here:

6e059a2

Please note also the following:

This requires some research about how to avoid too many reconciles + scale tests.

So its not just about start watching and emitting events for this pods.

[arshadd-b]

Thank @chrischdi sharing details

This requires some research about how to avoid too many reconciles + scale tests.

sure will check this