Usage of continuous in GitOps Principle Nr. 3

[murillodigital]

The following living document reflects the current state of definition around the core principles of the GitOps model https://hackmd.io/arwvV8NUQX683uBM3HzyNQ, as collaboratively defined by the community and the GitOps Principles Committee.

This discussion pertains Principle 3 and the use of the word continuous as a component of the title.

An argument has been raised as to the inaccuracy or impossibility of continuous being a determining factor of the GitOps reconciliation process, considering concerns such as incidents, freeze windows as well as other corner cases in which stating a fixed expectation of continuity may not reflect the true nature of production and enterprise systems.

Discussions surrounding this subject can be found in:

Mar 29, 2021

• Google Doc
• YouTube
• Zoom

April 05, 2021

• Google Doc
• YouTube
• Zoom

April 14, 2021

• Google Doc
• YouTube

Please use this discussion to communicate your position as succinctly as possible around the use of the word continuous in the principle, as well as the implications considering your own experience with as well as expectations around GitOps.

[moshloop]

My objection to the use of continuous is around the dictionary definition of uninterrupted and without gaps

1. Continuous is an impossible state for a distributed system, even the fastest reconciliation loops will happen in a few seconds and while not a large gap or interruption it is a gap nonetheless

   This distinction has already come back to bite the continuous delivery movement with a split into continuous deployment and continuous delivery (with continuously deliverable being the now accepted meaning)

   Even continuous integration falls into this trap (CI doesn't mean we are continuously building or running a daily build, it means the state of development is continuously integrateble (and code sitting on a developers workstation not being checked in, is not integrateble)

   A principle should be unassailable and have no room for interpretation - refuting the dictionary definition of a term, defeats the purpose of a principle - something that succinctly explains a concept or idea

2. Reconciliation loops MUST run continuously to check the state, but the reconciliation action cannot physically and should not run continuously, a PID based reconciliation loop (which is what you need to not shoot yourself in the foot) by design inserts gaps and interruptions based on the feedback from the system

3. Even if the continuous application of state is desirable (which control theory states it is not) it is not pragmatic, from my understanding the WG was brought together to increase the adoption levels of GitOps, you don't do that by raising the barrier to entry to a point that nothing in reality actually meets the criteria set forward.

We should take a step back and ask what are the current barriers to entry and how do we remove them, not adding new barriers.

[theWilliamChia]

@moshloop Everything you are saying makes complete sense to me. I'm not sure why GitOps couldn't simply have a principle of "state reconciliation" rather than "continuous state reconciliation".

This stood out to me

WG was brought together to increase the adoption levels of GitOps, you don't do that by raising the barrier to entry to a point that nothing in reality actually meets the criteria set forward.

On this topic, I'm curious, what if reconciliation was periodic, or even regular, rather than continuous? These seem like states of maturity along a GitOps continuum rather than a binary yes/no.

It doesn't seem helpful to tell someone, "No you're not doing GitOps, you need to be at this level of maturity before you are really doing GitOps."

It seems more helpful to say, "these are the core principles and if you have the kernel of those, that's GitOps, and oh by the way, we can help move you to the next level." That feels similar to how DevOps is.

[schlomo]

@moshloop thanks a lot for writing down your thoughts, now I understand much better the point that bothers you.

Do we agree on the fact that the checking for deviation should run continuously, your pint (2)? That was my motivation to wish for "continuous" to be part of the principle.

If I understand your arguments correctly, It seems to me that a wording like "The principle of continuous state reconciliatiatability" (or however in correct English) would be more appropriate in order to reflect - analog to CD meaning continuously deliverable - that it is in principle possible to reconcile the state at any given moment.

So as a compromise, could we somehow rephrase the principle to make it clear that the checking happens all the time - continuously - and that then the decision to reconcile includes also the option to not reconcile at that moment?

Software agents continuously, and automatically, compare a system’s Actual State to its Desired State. If the actual and desired states differ for any reason, automated actions to reconcile them are initiated.

The principle now has two sentences. The first one mentions continuous checking and the second implies immediate action. Would changing the second sentence to something like this better express what you meant?

If the actual and desired states differ for any reason, automated actions to reconcile them can be initiated.

Or is that already too weak language? What would you suggest?

About the heading of the principle: I think that if "continuously" stays in the first sentence then it can be omitted in the heading as the continuous checking is the important part in my opinion.

[moshloop]

On this topic, I'm curious, what if reconciliation was periodic, or even regular, rather than continuous? These seem like states of maturity along a GitOps continuum rather than a binary yes/no.

I would argue that most if not all reconciliation are already periodic and regular (e.g flux polls for commits every 5m by default) - Webhooks can speed up the loop to poll sooner

It doesn't seem helpful to tell someone, "No you're not doing GitOps, you need to be at this level of maturity before you are really doing GitOps."

Completely agree, we need to define a baseline e.g. Not reliant on webhooks or manually triggered jobs and then offer recommendations that increase maturity (but not on a mature scale)

Do we agree on the fact that the checking for deviation should run continuously, your pint (2)? That was my motivation to wish for "continuous" to be part of the principle.

100%

The principle of continuous state reconciliatiatability

I am not sure the *ability maps into GitOps, The state should always be reconciliable (through the use of linting and unit tests) but in addition, the checking should also run in a loop

Software agents continuously, and automatically, compare a system’s Actual State to its Desired State. If the actual and desired states differ for any reason, automated actions to reconcile them are initiated.

This description is ok as initiated doesn't imply completion - @todaywasawesome suggested automated actions attempt to reconcile them which I like because it implies that an attempt could fail e.g. for incidents, change freezes, etc..

What I would suggest is

3. The principle of state reconciliation loops
Software agents continuously, and automatically, compare a system’s Actual State to its Desired State. If the actual and desired states differ for any reason, automated actions attempt to reconcile them

• The desired state MUST be continuously reconciliable through the use of linting/unit tests
• The desired state MUST be continuously polled for changes (which can be accelerated through webhooks, but webhooks alone is not GitOps)
• The desired and actual state SHOULD be compared continuously to detect drift and feedback into the state store.
• The actual state SHOULD feedback into the state store (e.g. Github Deployment tracking the rollout status)
• The reconciliation loop SHOULD limit logic around policy (e.g. for change freeze periods should be implemented in merge blocking tests on pull requests)

[lotharschulz]

I wonder if a crisp principle title like The principle of state reconciliation is enough.

@moshloop's proposal principle description

Software agents continuously, and automatically, compare a system’s Actual State to its Desired State. If the actual and desired states differ for any reason, automated action attempts to reconcile them.

can be implemented with loops though. However other options might be possible as well.

[schlomo]

3. The principle of state reconciliation loops
Software agents continuously, and automatically, compare a system’s Actual State to its Desired State. If the actual and desired states differ for any reason, automated actions attempt to reconcile them

I am happy to agree with this version.

[lloydchang]

@moshloop wrote:

The reconciliation loop SHOULD limit logic around policy (e.g. for change freeze periods should be implemented in merge blocking tests on pull requests)

---

@moshloop ❓ Is the intent to discourage logic around policy such as the following? Thanks.

If there is an active matching allow and an active matching deny then syncs will be denied as deny windows override allow windows.
https://argo-cd.readthedocs.io/en/stable/user-guide/sync_windows/

[moshloop]

I am personally indifferent to whether this should be discouraged or not (as long as it is still allowed) others have a view that this should be done as part of the pull request workflow and the commits should only merge inside the change/sync period

[lloydchang]

LGTM

[moshloop]

can be implemented with loops though. However other options might be possible as well.

Could you elaborate on this? I am struggling to think of any other option?

[moshloop]

I think it is the other options like event-based systems that we want to exclude, and adding in the term loop achieves that

[lotharschulz]

I see your point and I agree with our discussion as context. Without our discussion as context I had a hard time to derive a conclusion like "adding in the term loop excludes event-based systems". What if we explicitly describe that event-based systems are excluded? Such a description could be part of a principles notes section.

[lloydchang]

@bricef @moshloop @lotharschulz

Are you suggesting to exclude the following?

• The Notification Controller is a Kubernetes operator, specialized in handling inbound and outbound events.
• The controller handles events coming from external systems (GitHub, GitLab, Bitbucket, Harbor, Jenkins, etc) and notifies the GitOps toolkit controllers about source changes.
• The controller handles events emitted by the GitOps toolkit controllers (source, kustomize, helm) and dispatches them to external systems (Slack, Microsoft Teams, Discord, Rocker) based on event severity and involved objects.
https://toolkit.fluxcd.io/components/notification/controller/

[moshloop]

I think the outbound events (unless they are back to the git repository) are supplementary and neither included nor excluded.

If the notification controller is the only trigger for state reconciliation, then yes it would be excluded, it can be in addition to a polling reconciliation, but not instead of.

What if we explicitly describe that event-based systems are excluded? Such a description could be part of a principles notes section.

I agree we should elaborate more in the description,

[lloydchang]

@moshloop wrote:

If the notification controller is the only trigger for state reconciliation, then yes it would be excluded, it can be in addition to a polling reconciliation, but not instead of.

My understanding is the notification controller's webhook receivers are additions, based on the following written by @stefanprodan:

Webhook receivers
GitOps controllers are by nature pull based, in order to notify the controllers about changes in Git or Helm repositories, one may wish to setup webhooks and trigger a cluster reconciliation every time a source changes.

Setup Webhook Receivers
The GitOps toolkit controllers are by design pull-based. In order to notify the controllers about changes in Git or Helm repositories, you can setup webhooks and trigger a cluster reconciliation every time a source changes. Using webhook receivers, you can build push-based GitOps pipelines that react to external events.

Thank you @moshloop @stefanprodan @lotharschulz @bricef

[lloydchang]

@murillodigital wrote:

Discussions surrounding this subject can be found in the following recordings: here and here (link coming soon).

@murillodigital When your time permits, please amend the above as follows:

Discussions are in:

Mar 29, 2021
• Google Doc
• YouTube
• Zoom

April 05, 2021
• Google Doc
• YouTube
• Zoom

April 14, 2021
• Google Doc
• YouTube

Thank you!

[scottrigby]

Added, thanks for the reminder

[lloydchang]

Thanks @theWilliamChia for clarifying that the link was outdated.

Thanks @scottrigby for adding additional links.

[scottrigby]

For posterity, the Principles Committee has since agreed to keep "continuous" as long as we keep a prominent note about this colocated with the principles. As of v0.1.0 of the OpenGitOps documents: https://github.com/open-gitops/documents/blob/v0.1.0/PRINCIPLES.md this was addressed (see Continuous glossary item).