Is the Repo per app scenario possible with gitops-connector?

[cyberjpb1]

Is the Repo per app scenario possible with gitops-connector?
Currently I have a repo for each application which also contains the manifests.

If the answer to my question is yes, how to configure it?
gitRepositoryType: AZDO
ciCdOrchestratorType: AZDO
gitOpsOperatorType: FLUX
azdoGitOpsRepoName:
azdoOrgUrl:
azdoPrRepoName:
gitOpsAppURL:
orchestratorPAT:

[eedorenko]

You need to have multiple instances of gitops-connector on your clusters. One instance per application/repo

[markphillips100]

@eedorenko would a k8 operator approach be feasible here? To clarify, I mean one installation of gitops-connector (operator + custom Connection CRD) which then handles multiple subscribe-to-notification/publish-to-specific-repo Connection resources.

[markphillips100]

@eedorenko I've added support for multiple configs in the one instance via a CRD if you are interested? It still supports the original env config via helm values type of setup, albeit it is a breaking change due to the values restructuring. The switch between modes of operation are determined by setting singleInstance: null or supplying values. All is explained in the helm chart readme.

Whilst it works for ArgoCD notifications as-is, I haven't tested with flux as my environment isn't setup to deal with it. It's not a great deal of work and can explain if this goes further.

My fork is here, and let me know if you want a PR opened.

[cyberjpb1]

@markphillips100 Very interesting, I will try your approach by doing a test with Flux. I will get back to you as soon as possible.

[markphillips100]

@cyberjpb1 For the flux_gitops_operator to filter supported messages and indicate its support for the required config name in phase_data, the is_supported_message function needs fleshing out here. See the argo_gitops_operator change for how I implemented it in that use case.

I imagine in FluxV2 use case we would need to make use of the Alert's eventMetadata to convey the required config name in the phase_data, although being unfamiliar with Flux I do not know how this results in the phase_data structurally to know what to look for.

[markphillips100]

@cyberjpb1 I checked a previous PR you opened for insight into the eventMetadata and that gave me enough info to create the is_supported_operator. I've created a new flux-multi-config-support branch with this change.

So in theory, the following should suffice:

1. add a gitops_connector_config_name: "<name of config>" to the Alert's eventMetadata,
2. set singleInstance: null in values.yaml,
3. apply a gitopsconfig manifest to the cluster where gitops-connector is running - ensure the name is the same name used in step 1.

NOTE: The helm chart creates a service account, role and role binding to support the connector watching and updating the gitopsconfig resource. The operator also automatically patches (hence the updating) a finalizer into the resource to ensure when it is deleted that a proper cleanup occurs before the manifest is removed from the cluster.

[cyberjpb1]

@markphillips100 Hello, sorry for the delay of my response but my schedule did not allow me to do a test before today.
I tried to follow your installation procedure but it does not work, I must be missing something in the configuration.

I'm using Red Hat CodeReady Containers (crc) for my tests.

Here is the log when the pod starts.

[2024-11-24 00:21:37 +0000] [7] [INFO] Starting gunicorn 20.0.4
[2024-11-24 00:21:37 +0000] [7] [INFO] Listening at: http://0.0.0.0:8080 (7)
[2024-11-24 00:21:37 +0000] [7] [INFO] Using worker: sync
[2024-11-24 00:21:37 +0000] [8] [INFO] Booting worker with pid: 8
DEBUG:root:Detected no ENV configuration data. Running in multiple instance configuration mode via gitopsconfig resources.
INFO:root:Starting Kopf operator thread
INFO:root:Starting GitOps Operator
DEBUG:asyncio:Using selector: EpollSelector
/usr/local/lib/python3.9/site-packages/kopf/_core/reactor/running.py:179: FutureWarning: Absence of either namespaces or cluster-wide flag will become an error soon. For now, switching to the cluster-wide mode for backward compatibility.
warnings.warn("Absence of either namespaces or cluster-wide flag will become an error soon."
DEBUG:kopf._core.reactor.running:Starting Kopf 1.37.3.
INFO:kopf._core.engines.activities:Initial authentication has been initiated.
DEBUG:kopf.activities.authentication:Activity 'login_via_client' is invoked.
WARNING:kopf._core.reactor.running:OS signals are ignored: running not in the main thread.
DEBUG:kopf.activities.authentication:Client is configured in cluster with service account.
INFO:kopf.activities.authentication:Activity 'login_via_client' succeeded.
INFO:kopf._core.engines.activities:Initial authentication has finished.
WARNING:kopf._core.reactor.observation:Non-patchable resources will not be served: {gitopsconfigs.v1.apps-crc.testing}
DEBUG:kopf._cogs.clients.watching:Starting the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
WARNING:kopf._core.reactor.observation:Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts.

[markphillips100]

The last warning in your logs would point at some role/role-binding issue for the service account the operator is configured to run with. Maybe take a look there to ensure the watch permission is available on gitopsconfigs crd.

[markphillips100]

The helm chart should be setting that up for you here

[cyberjpb1]

Yes, I use your Helm Chart, the only thing I changed is replacing "example.com" by "apps-crc.testing".

[markphillips100]

Yeah was noticing that but wouldn't think it would be a problem....prior to the edit :-)

[markphillips100]

Did you by any chance install the helm chart prior to making the api group changes? Just wondering if there are now 2 CRDs for the same Kind.

The 3 event handlers here aren't currently checking for apiVersion so would be just chance which one is used if more than one CRD group exists. These handlers would need to change (and probably should anyway) to include the group name.

[cyberjpb1]

It's as if the callback was never sent.
The task in the pipeline wait indefinitely.

[markphillips100]

DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
WARNING:kopf._core.reactor.observation:Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts.

Can you delete the operator pod (without uninstalling/reinstalling helm chart) and confirm if logs still show this warning please? It's possibly a race condition re permissions not set by time pod comes up so a restart of pod should negate that.

[cyberjpb1]

OK, I deleted the pod and here is the log of the new pod:

[2024-11-24 02:36:45 +0000] [7] [INFO] Starting gunicorn 20.0.4
[2024-11-24 02:36:45 +0000] [7] [INFO] Listening at: http://0.0.0.0:8080 (7)
[2024-11-24 02:36:45 +0000] [7] [INFO] Using worker: sync
[2024-11-24 02:36:45 +0000] [8] [INFO] Booting worker with pid: 8
DEBUG:root:Detected no ENV configuration data. Running in multiple instance configuration mode via gitopsconfig resources.
INFO:root:Starting Kopf operator thread
INFO:root:Starting GitOps Operator
DEBUG:asyncio:Using selector: EpollSelector
/usr/local/lib/python3.9/site-packages/kopf/_core/reactor/running.py:179: FutureWarning: Absence of either namespaces or cluster-wide flag will become an error soon. For now, switching to the cluster-wide mode for backward compatibility.
warnings.warn("Absence of either namespaces or cluster-wide flag will become an error soon."
DEBUG:kopf._core.reactor.running:Starting Kopf 1.37.3.
INFO:kopf._core.engines.activities:Initial authentication has been initiated.
DEBUG:kopf.activities.authentication:Activity 'login_via_client' is invoked.
WARNING:kopf._core.reactor.running:OS signals are ignored: running not in the main thread.
DEBUG:kopf.activities.authentication:Client is configured in cluster with service account.
INFO:kopf.activities.authentication:Activity 'login_via_client' succeeded.
INFO:kopf._core.engines.activities:Initial authentication has finished.
WARNING:kopf._core.reactor.observation:Non-patchable resources will not be served: {gitopsconfigs.v1.apps-crc.testing}
DEBUG:kopf._cogs.clients.watching:Starting the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
DEBUG:kopf._cogs.clients.watching:Stopping the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
WARNING:kopf._core.reactor.observation:Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts.

[markphillips100]

So still a permission issue by the looks, preventing the operator watching for the gitopsconfig. I would expect to see some DEBUG lines pertaining to the GitopsConnectorManager setting up a GitopsConnector for the name supplied in your GitOpsConfig manifest.

The other messages in the previous log are just the raw event data coming from flux via the /gitopsphase endpoint. They get ignored because no GitopsConnecter is configured basically.

Any chance you can revert to a clean install of the original example.com CRD just so we can rule that out completely?

[cyberjpb1]

OK, I'll try that tomorrow.
Thanks a lot for your help, it's much appreciated.

[markphillips100]

No problem

[cyberjpb1]

You were right, it was an old version of the GitOpsConfig that was stuck.
I had to do an "oc replace" command to unstick it and be able to delete it.

Now I have this that keeps coming back in a loop in the log (I have hidden the sensitive values.)

DEBUG:root:_should_update_abandoned_pr. should_update: False
DEBUG:root:_should_update_abandoned_pr called. pr_data: {
"repository": {
"id": "99999999-9999-9999-9999-999999999999",
"name": "WebAppExemple2.IaC",
"url": "https://dev.azure.com/myorganization/99999999-9999-9999-9999-999999999999/_apis/git/repositories/99999999-9999-9999-9999-999999999999",
"project": {
"id": "99999999-9999-9999-9999-999999999999",
"name": "GitOps",
"state": "unchanged",
"visibility": "unchanged",
"lastUpdateTime": "0001-01-01T00:00:00"
}
},
"pullRequestId": 285,
"codeReviewId": 285,
"status": "abandoned",
"createdBy": {
"displayName": "GitOps Build Service (myorganization)",
"url": "https://spsprodcca1.vssps.visualstudio.com/99999999-9999-9999-9999-999999999999/_apis/Identities/99999999-9999-9999-9999-999999999999",
"_links": {
"avatar": {
"href": "https://dev.azure.com/myorganization/_apis/GraphProfile/MemberAvatars/svc.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}
},
"id": "99999999-9999-9999-9999-999999999999",
"uniqueName": "Build\99999999-9999-9999-9999-999999999999",
"imageUrl": "https://dev.azure.com/myorganization/_api/_common/identityImage?id=99999999-9999-9999-9999-999999999999",
"descriptor": "svc.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
"creationDate": "2024-07-29T18:18:33.2230957Z",
"closedDate": "2024-07-29T19:03:55.5257799Z",
"title": "Deploiement Helm Chart 904.0.0 en dev sur testing",
"sourceRefName": "refs/heads/devRelease-904.0.0",
"targetRefName": "refs/heads/master",
"isDraft": false,
"mergeId": "99999999-9999-9999-9999-999999999999",
"lastMergeSourceCommit": {
"commitId": "99999999-9999-9999-9999-999999999999",
"url": "https://dev.azure.com/myorganization/99999999-9999-9999-9999-999999999999/_apis/git/repositories/99999999-9999-9999-9999-999999999999/commits/99999999-9999-9999-9999-999999999999"
},
"lastMergeTargetCommit": {
"commitId": "99999999-9999-9999-9999-999999999999",
"url": "https://dev.azure.com/myorganization/99999999-9999-9999-9999-999999999999/_apis/git/repositories/99999999-9999-9999-9999-999999999999/commits/99999999-9999-9999-9999-999999999999"
},
"reviewers": [],
"url": "https://dev.azure.com/myorganization/99999999-9999-9999-9999-999999999999/_apis/git/repositories/99999999-9999-9999-9999-999999999999/pullRequests/285",
"completionOptions": {
"squashMerge": true,
"mergeStrategy": "squash"
},
"supportsIterations": true
}
DEBUG:root:_should_update_abandoned_pr. should_update: False

[markphillips100]

This is the abandoned PR status reconciliation which is the same as the original gitops-connector code. At least I don't recall making changes here other than more logging.

_should_update_abandoned_pr. should_update: False

This statement indicates the PR was closed (abandoned) longer than the constant 72 hours defined in code so the processing just ignores further processing of the PR's status. Unfortunately there is no way in Azure Devops to delete abandoned PRs.