Should a WIP pact be non-pending once successfully verified?

[bethesque]

Current flow/problem

1. Provider is configured to verify main/test/prod pacts and has pending enabled and includes WIP pacts since the start of the year.
2. Consumer creates feature branch, pushes feature pact with new interaction.
3. Provider makes a feature branch, build runs for feature branch, verifies main/test/prod pacts and the new feature pact, which is brought in as WIP.
   • Feature pact is in pending state. Feature pact verification fails, but build is green because it is pending.
4. Provider makes changes so that they now implement the features described in the feature pact.
5. Provider pushes, CI runs for feature branch, feature pact is successfully verified.
   • Feature pact is still WIP (as it has not been successfully verified while NOT wip yet).
   • Feature pact is still in pending state, but only because WIP pacts are hardcoded to be pending=true. If we used the actual pending calculation, it would no longer be pending.
6. Provider makes some other changes that unintentionally break the feature pact. The build does not fail, because the feature pact was in (hardcoded) pending state.
7. Provider team does not notice that the feature pact is broken, because the build is green, and merges branch into master.

Proposal - the pending state for WIP pacts should be dynamically calculated, not hardcoded to true, so that you can't accidentally make a regression to a passing WIP pact. The impact of this would be that WIP pacts could now fail the build (where as currently they can't).

Discuss!

[ertrzyiks]

I think the scenario you described for a feature branch of the provider actually makes sense to cover 👍 I'm concerned about the main branch pipeline.

Do I get this diagram right? https://docs.pact.io/pact_nirvana/step_4

CI/CD pipeline includes Verify Pacts step, which with the change you propose could potentially fail in case that the provider version is otherwise safe to release just because it breaks a backward compatibility with a future (WIP) version of a consumer.

Another scenario I see problematic: we have a WIP tag of the consumer which was successfully verified and selected so it became WIP but no longer pending. What if the consumer contract is changed and provider is no longer satisfying the requirements? Would such WIP consumer tag break all the verifications or switch back to pending?

[bethesque]

the change you propose could potentially fail in case that the provider version is otherwise safe to release just because it breaks a backward compatibility with a future (WIP) version of a consumer.

Yes - this would be the trade off/impact. If you made a WIP pact pass on master, but then decided not to support it and reverted the change, the master build would break. I can't think of a convincing scenario for why someone might do this though.

[bethesque]

What if the consumer contract is changed and provider is no longer satisfying the requirements?

The consumer branch would revert to pending, because the WIP calculation is based on the pact content of the latest pact for each branch. The pact content that was previously the latest for its branch is now no longer the latest for its branch - there is a new content, and it would go through the same lifecycle as before.

[PhilHardwick]

It's a tough choice because you have the option of making the pending calculation more complicated or keeping it simple but putting the onus on the provider team to notice when things are failing or to fix their build after they merge a regression.

CI/CD pipeline includes Verify Pacts step, which with the change you propose could potentially fail in case that the provider version is otherwise safe to release just because it breaks a backward compatibility with a future (WIP) version of a consumer.

To overcome this I think the pending state would have to be reset to true if the provider was verifying on a new tag - as in the change to pending=false would be scoped to just the tag (the feature branch) the provider was verifying from. We don't currently "use" the provider tags (tbh I'm not 100% on what they're used for) we always have it set to our main branch, but this would mean we need to start setting it correctly, locally and on PR builds.

[bethesque]

The pending status is always calculated on a per-provider-branch basis. Though if you don't set any tags, it would be a global status.

you have the option of making the pending calculation more complicated or keeping it simple

It's only simple for wip pacts. For pacts that are explicitly specified with tags/selectors, there is already a more complicated calculation behind it (working out if there has already been a successful verification from that provider branch or not). At the moment, it's just not used for WIP pacts.

[solarmosaic-kflorence]

The failing of the build would be gated by tag when provider tags are used, correct? In other words, if the consumer and provider are both using tag feat-x and the pact is verified by the provider, it would then become a non-WIP pact, and so the build would fail if the pact was broken (I believe this is existing functionality). So the above would only have an impact in the scenario where the provider is not explicitly using the same tag as the consumer? In that case, once the provider build verifies and is tagged with master, the potentially still WIP consumer pact could cause provider builds in master to fail if they break that pact. If that's the case, I would prefer to just leave it as "the provider must explicitly opt-in to allowing non-stable (WIP) consumer pacts to fail the build".

[solarmosaic-kflorence]

Although perhaps I am confused by the WIP feature as it exists now, and what I describe above (as "existing functionality") would be unlocked by your original statement, in which case, I would intuitively want that.

[bethesque]

I added this to the docs the other day.

"WIP" (work in progress) refers to the reason the pact is being verified. Either a pact is being verified because you specified a selector for it, or it is being verified because it has been determined to be a "work in progress" pact. "WIP" and "explicitly selected" are mutually exclusive reasons for a pact to be verified.

"Pending" is a property of a pact that is being verified, and it exists on both WIP and explicitly selected pacts.

For the explicitly selected pacts, the pending flag is calculated dynamically, based on the provider branch (which is represented as a tag). If there are no successful verifications from this branch, then it is pending. Once there is a successful verification from that branch, it is no longer pending.

For the WIP pacts, the pending flag is hardcoded to true.

Hopefully that makes it clear.

There's no special branch matching logic going on for WIP pacts. It's just that many people do use the branch name in the selector, and that means that the matching pact is verified because it was explicitly specified, and not because it was determined to be WIP.

Having thought this through, I'm convinced that we should leave it like it is. If someone is working on a feature branch for a specific pact, and they want it to fail if they accidentally break it, then they can update the selectors to include that pact.