Use the last transition time to track and assess status conditions

[leochr]

The unchangedConditionCount field in the status condition can interfere with updating CR, especially at the initial stage when the reconcile interval is 5 seconds. Eventually, it'll only be updated every 120 seconds, but can still be disruptive.

We should consider using the lastTransitionTime instead (if appropriate, otherwise lastUpdateTime) to determine when to increase the reconcileInterval. In some cases, the status condition and lastTransitionTime are not set/updated correctly, so that must be resolved first.

If the above is possible, then we should also support a configuration option to disable .status.reconcileInterval field (i.e. reconcileIntervalHide), so that users are not interrupted by status updates when trying to update CR spec.

[halim-lee]

The tests were run on a large OCP cluster. Average of 10 test sets.

After the code changes:

• All instances in a single namespace tests:

  • It took 4m50s to fully reconcile 100 instances, all in the same namespace.
  • It took 30s to fully reconcile 1 new instance, with 100 working instances.
  • It took 30s to fully reconcile 1 incorrectly configured instance after it was corrected, with 101 working instances and 9 non-working instances running.

• Each instance in its own namespace tests:

  • It took 9m21s to fully reconcile 100 instances, each in its own namespace.
  • It took 1m7s minutes to fully reconcile 1 new instance, with 100 working instances.
  • It took 2m29s minutes to fully reconcile 1 wrongly configured instance after it was corrected, with 101 working instances and 9 non-working instances running.

Before the code changes:

• All instances in a single namespace tests:

  • It took 5m11s to fully reconcile 100 instances, all in the same namespace.
  • It took 2m4s to fully reconcile 1 new instance, with 100 working instances.
  • It took 1m19s to fully reconcile 1 wrongly configured instance after it was corrected, with 101 working instances and 9 non-working instances running.

• Each instance in its own namespace tests:

  • It took 10m to fully reconcile 100 instances, each in its own namespace.
  • It took 2m17s minutes to fully reconcile 1 new instance, with 100 working instances.
  • It took 2m29s minutes to fully reconcile 1 wrongly configured instance after it was corrected, with 101 working instances and 9 non-working instances running.