[backport v2.10] [SURE-9460] Fleet not picking up gitrepo updates, no job created to update

[rancherbot]

This is a backport issue for #3138, automatically created via GitHub Actions workflow initiated by @manno

Original issue body:

SURE-9460

Issue description

After upgrading Rancher to 2.9.3 / fleet to v0.10.4, some gitrepos are no longer receiving updates. Customer update the repository, but changes are not pushed to the clusters. No Job is created to pull in the changes that should be tracked by the gitRepo.

In fleet v0.10.4, there were changes made to how jobs are managed in fleet. Could these changes be the cause of the issue here? #2932 seems to change how jobs are managed.

Business impact:

Unable to receive updates to applications using fleet for continuous delivery.

Troubleshooting steps:

GitJob pod, does not show that jobs are completing for those gitRepos, We are also unable to find jobs for the

Repro steps:

Upgrade to Rancher 2.9.3 from 2.9.2

Workaround:

Is a workaround available and implemented? yes
What is the workaround:
Customer found that by editing a gitRepo in the Rancher UI, changing nothing, then saving, it will eventually cause the repo to pull the change and make the necessary updates.

When making those changes, a couple lines are changed within the gitRepo:
spec.correctDrift: {} is added
status.commit is updated
status.lastPollingTriggered time is updated (time changed by more than a day).

Actual behavior:

repositories are not updated.

Expected behavior:

Repositories are updated.

[mmartin24]

Checked jitter addition the following way

• Add debug mode for fleet
• Add gitrepo setting pollingInterval to 60s
• Check the logs on the gitjob pod filtering for Reconciling Gitrepos status
• Verify that it polls the gitrepo with 60 seconds + small range (I noticed between 1 and 6 seconds)

Observed timestamps in logs with fix (v2.10-ee01a8e72c881512d9d2352229a46fcea7c9d8e8-head with fleet 105.0.3+up0.11.3-rc.2. Here we see timestamps are 60 seconds + range (63,65,66,...)

Logs in version WITH Jitter

2025-01-15T16:47:11Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "fe7178a7-e2bd-4d39-9fec-0ed24a105ec2", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:48:13Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "404ba13c-b637-46f0-abc4-3a0be153249c", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:49:16Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "31e2391c-f1cd-4a19-8fb0-adca27172f16", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:50:17Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "9f5c46a9-0bff-4f45-b652-8896660806b4", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:51:20Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "31a86c97-579b-4d8d-8467-75040408461d", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:52:23Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "94757b08-fb53-4838-a553-1120e645417f", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:53:28Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "b80825ab-04c0-4a6c-9b60-b484c49a9f9e", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:54:31Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "4f88fb71-161e-4a33-8387-97ce450b3f8f", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}
2025-01-15T16:55:34Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-jitter","namespace":"fleet-default"}, "namespace": "fleet-default", "name": "test-jitter", "reconcileID": "a1886137-46f0-4ce0-add9-83555df1f93b", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T16:32:52Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:32:23Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:32:23Z"}]}

Observed timestamps in logs with fix (v2.10.0 with fleet 1105.0.1+up0.11.1) . Here timestamps occur every 60 seconds

Logs in version WITHOUT Jitter

2025-01-15T20:55:06Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "6878d650-cc7f-461b-8a1b-49ec5c635b24", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T20:56:07Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "dd85a260-0b97-453e-8144-9a921f8b3b52", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T20:57:07Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "de389d7a-ef7d-455a-a434-8df152620dbf", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T20:58:08Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "2cfda620-da9c-4e3b-acde-4cae6de857ea", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T20:59:08Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "3417737b-7cf9-40bd-a741-e00db3115c27", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}
2025-01-15T21:00:09Z    DEBUG   gitops-status   Reconciling GitRepo status      {"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-pre-jitter","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-pre-jitter", "reconcileID": "8dc8b542-cee0-427d-99f1-db91b13af96e", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-15T17:00:06Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-15T16:59:49Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-15T16:59:49Z"}]}

It seems to be working for me.

Verified as well GITREPO_SYNC_PERIOD default set to 2 hours in v2.10-ee01a8e72c881512d9d2352229a46fcea7c9d8e8-head with fleet 105.0.3+up0.11.3-rc.2:

   spec:
      containers:
      - args:
        - fleetcontroller
        - gitjob
        - --gitjob-image
        - rancher/fleet:v0.11.3-rc.2
        - --debug
        - --debug-level
        - "1"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CATTLE_ELECTION_LEASE_DURATION
          value: 30s
        - name: CATTLE_ELECTION_RETRY_PERIOD
          value: 10s
        - name: CATTLE_ELECTION_RENEW_DEADLINE
          value: 25s
        - name: GITREPO_SYNC_PERIOD
          value: 2h

When pollingInterval is set to 5 minutes and GITREPO_SYNC_PERIOD is set to 1 minute, logs for reconciliation are triggered every minute approximately as can be seen in the gitjob log when filtering for Reconciling Gitrepo status:

k logs -n cattle-fleet-system gitjob-5f8956656b-5h6pq -f | grep "Reconciling GitRepo status"

Output:

2025-01-16T14:11:16Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-gitrepo-sync-period","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-gitrepo-sync-period", "reconcileID": "57889cc7-c99c-445e-a02a-7d53146f82d2", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-16T14:04:41Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"}]}
2025-01-16T14:12:12Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-gitrepo-sync-period","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-gitrepo-sync-period", "reconcileID": "964f578a-e16b-4ff2-961c-b42f2371fc6a", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-16T14:04:41Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"}]}
2025-01-16T14:13:07Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-gitrepo-sync-period","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-gitrepo-sync-period", "reconcileID": "96e3fac0-84bd-4fe9-9191-637e1fa2e7a3", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-16T14:04:41Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"}]}
2025-01-16T14:14:03Z	DEBUG	gitops-status	Reconciling GitRepo status	{"controller": "GitRepoStatus", "controllerGroup": "fleet.cattle.io", "controllerKind": "GitRepo", "GitRepo": {"name":"test-gitrepo-sync-period","namespace":"fleet-local"}, "namespace": "fleet-local", "name": "test-gitrepo-sync-period", "reconcileID": "7ee1e3d9-2cf8-473c-b00e-6c5c44a84c17", "generation": 2, "commit": "dcf4917293ef131f64724d0c03cadc4f5b257168", "conditions": [{"type":"Ready","status":"True","lastUpdateTime":"2025-01-16T14:04:41Z"},{"type":"GitPolling","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Reconciling","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Stalled","status":"False","lastUpdateTime":"2025-01-16T14:04:34Z"},{"type":"Accepted","status":"True","lastUpdateTime":"2025-01-16T14:04:34Z"}]}

So GITREPO_SYNC_PERIODseems working as well