[SURE-9460] Fleet not picking up gitrepo updates, no job created to update

[kkaempf]

SURE-9460

Issue description

After upgrading Rancher to 2.9.3 / fleet to v0.10.4, some gitrepos are no longer receiving updates. Customer update the repository, but changes are not pushed to the clusters. No Job is created to pull in the changes that should be tracked by the gitRepo.

In fleet v0.10.4, there were changes made to how jobs are managed in fleet. Could these changes be the cause of the issue here? #2932 seems to change how jobs are managed.

Business impact:

Unable to receive updates to applications using fleet for continuous delivery.

Troubleshooting steps:

GitJob pod, does not show that jobs are completing for those gitRepos, We are also unable to find jobs for the

Repro steps:

Upgrade to Rancher 2.9.3 from 2.9.2

Workaround:

Is a workaround available and implemented? yes
What is the workaround:
Customer found that by editing a gitRepo in the Rancher UI, changing nothing, then saving, it will eventually cause the repo to pull the change and make the necessary updates.

When making those changes, a couple lines are changed within the gitRepo:
spec.correctDrift: {} is added
status.commit is updated
status.lastPollingTriggered time is updated (time changed by more than a day).

Actual behavior:

repositories are not updated.

Expected behavior:

Repositories are updated.

[carneiroskeeled]

@kkaempf have you tried to enable this?

  #  correctDrift:
  #    enabled: false

Or with the webhooks?

[manno]

We suspect that the GitRepo drops out of the new polling. The new polling is based on RequeueAfter and uses the reconcilers workqueue.

We suspect this could happen, because of

• a condition prevents the resource from being queued, when it should
• resource drops out of requeue because of an unhandled error
• not enough reconcile workers, so that some reconciles get delayed indefinitely

Until we can reproduce this:
We are adding some jitter to the polling, so that 300 gitrepos don't reconcile at the same time.
We shorten the resync period to pick up dropped gitrepos.

[manno]

/backport v2.10.2

[mmartin24]

Checked jitter addition the following way

• Add debug mode for fleet
• Add gitrepo setting pollingInterval to 60s
• Check the logs on the gitjob pod filtering for Reconciling Gitrepos status
• Verify that it polls the gitrepo with 60 seconds + small range (I noticed between 1 and 6 seconds)

Observed timestamps in logs with fix (v2.10-ee01a8e72c881512d9d2352229a46fcea7c9d8e8-head with fleet 105.0.3+up0.11.3-rc.2. Here we see timestamps are 60 seconds + range (63,65,66,...)

Logs in version WITH Jitter

2025-01-15T16:47:11Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "fe7178a7-e2bd-4d39-9fec-0ed24a105ec2", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:48:13Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "404ba13c-b637-46f0-abc4-3a0be153249c", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:49:16Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "31e2391c-f1cd-4a19-8fb0-adca27172f16", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:50:17Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "9f5c46a9-0bff-4f45-b652-8896660806b4", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:51:20Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "31a86c97-579b-4d8d-8467-75040408461d", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:52:23Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "94757b08-fb53-4838-a553-1120e645417f", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:53:28Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "b80825ab-04c0-4a6c-9b60-b484c49a9f9e", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:54:31Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "4f88fb71-161e-4a33-8387-97ce450b3f8f", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:55:34Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "a1886137-46f0-4ce0-add9-83555df1f93b", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}

Observed timestamps in logs with fix (v2.10.0 with fleet 1105.0.1+up0.11.1) . Here timestamps occur every 60 seconds

Logs in version WITHOUT Jitter

2025-01-15T20:55:06Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "6878d650-cc7f-461b-8a1b-49ec5c635b24", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T20:56:07Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "dd85a260-0b97-453e-8144-9a921f8b3b52", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T20:57:07Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "de389d7a-ef7d-455a-a434-8df152620dbf", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T20:58:08Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "2cfda620-da9c-4e3b-acde-4cae6de857ea", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T20:59:08Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "3417737b-7cf9-40bd-a741-e00db3115c27", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T21:00:09Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "8dc8b542-cee0-427d-99f1-db91b13af96e", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}

It seems to be working for me.
@manno , please let me know if there is anything else observable qa wise. Otherwise, I will close the item.

[mmartin24]

Verified as well GITREPO_SYNC_PERIOD default set to 2 hours in v2.10-ee01a8e72c881512d9d2352229a46fcea7c9d8e8-head with fleet 105.0.3+up0.11.3-rc.2:

   spec:
      containers:
      - args:
        - fleetcontroller
        - gitjob
        - --gitjob-image
        - rancher/fleet:v0.11.3-rc.2
        - --debug
        - --debug-level
        - "1"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CATTLE_ELECTION_LEASE_DURATION
          value: 30s
        - name: CATTLE_ELECTION_RETRY_PERIOD
          value: 10s
        - name: CATTLE_ELECTION_RENEW_DEADLINE
          value: 25s
        - name: GITREPO_SYNC_PERIOD
          value: 2h

When pollingInterval is set to 5 minutes and GITREPO_SYNC_PERIOD is set to 1 minute, logs for reconciliation are triggered every minute approximately as can be seen in the gitjob log when filtering for Reconciling Gitrepo status:

k logs -n cattle-fleet-system gitjob-5f8956656b-5h6pq -f | grep "Reconciling GitRepo status"

Output:

2025-01-16T14:11:16Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-gitrepo-sync-period","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-gitrepo-sync-period", "reconcileID": "57889cc7-c99c-445e-a02a-7d53146f82d2", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-16T14:04:41Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"}]}
2025-01-16T14:12:12Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-gitrepo-sync-period","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-gitrepo-sync-period", "reconcileID": "964f578a-e16b-4ff2-961c-b42f2371fc6a", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-16T14:04:41Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"}]}
2025-01-16T14:13:07Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-gitrepo-sync-period","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-gitrepo-sync-period", "reconcileID": "96e3fac0-84bd-4fe9-9191-637e1fa2e7a3", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-16T14:04:41Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"}]}
2025-01-16T14:14:03Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-gitrepo-sync-period","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-gitrepo-sync-period", "reconcileID": "7ee1e3d9-2cf8-473c-b00e-6c5c44a84c17", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-16T14:04:41Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"}]}

So GITREPO_SYNC_PERIODseems working as well

[mmartin24]

I commented and closed accidentally on this issue after confusing it with the backport item meant for 2.10.2.(here)
To be rechecked in 2.11 after further updates.