Some reflections about how to improve the ZEP process

[rabernat]

I'll offer some reflections on how we might improve our process around ZEPs, spec evolution, etc. Currently we do it something like this

• Propose a new idea for a ZEP as a discussion on an issue
• Make a PR to this repo of the draft ZEP (e.g. Draft ZEP 0007: Strings #47)
• Discuss a bit in that issue
• Then the ZEP is merged, moved to a draft, and that discussion closes
• The discussion then moves elsewhere? But where? It's hard to tell? Maybe there is a discussion forum topic. Maybe a PR to the spec repo
• Then there is a vote maybe? This happens on a different issue?

At this point we have created half a dozen different issues / PRs etc to discuss the same feature. It's hard to keep track of. We lose momentum.

Here's what I'd like to see in the next iteration of this process.

• One single repo where all discussion about ZEPs, specs, extensions, etc. happens
• One PR for every new extension / ZEP. All discussion happens here.
• When the PR is merged, the ZEP has been accepted.
• [maybe] Eliminate the ZEPs website and just put the information directly in GitHub as markdown files
• Better documentation about how to navigate the process. For example...is every extension a ZEP?

[jbms]

This all sounds perfect to me.

[d-v-b]

Thumbs up from me

[rabernat]

@MSanKeys963 - do you think you would be able to implement these changes?

[normanrz]

I like the proposal. Just one question:

When the PR is merged, the ZEP has been accepted

Is that accepted or finalized?

[rabernat]

Is that accepted or finalized?

Is this distinction really important? TBH I've always found it confusing. I think the ZEP process has too many steps.

[MSanKeys963]

Thanks for opening the issue, @rabernat.

After finalising ZEP1 and ZEP2 and holding discussions in a couple of ZEP meetings (11/2 and 11/16), we realised that the ZEP process needs some changes.

Based on the previous ZEP meetings and discussions, I'm working on a proposal to redefine the existing ZEP process. A few things I'm focusing on in the revised ZEP process are:

• Progression of a ZEP proposal towards finalisation in a phased manner, which ensures participation from the implementation council
• Focusing on the actual software implementation of the ZEP and allocating a timeline for feedback/changes based on implementation before a ZEP is frozen/goes for voting
• Introducing a fast-track/lean ZEP type, which allows the addition of minor changes to the specification without going through the whole process, e.g. Add zstd codec zarr-specs#256

[joshmoore]

For what it's worth, the above for me is the ideal best case workflow, and I agree that this is the case for most changes recently. I think there are drawbacks for long and drawn out discussions like ZEP1 though, namely:

• for authors in the present, it can be very hard to deal with the many micro discussions
• for readers in the future, it's very hard to understand the reasoning of decisions

All that being said, I'm still very much for simplifying ZEP0 (and/or replacing with ezZEP -- get it? 😄), but I wonder if we can find a balance of the extremes. One issue not discussed above nor really in ZEP0 pertains to "breakingness". For pure additions / optional items, is a ZEP (even an ez one) simply overkill?

[MSanKeys963]

I want to expand on the revision I envision and gather feedback from everyone before moving ahead with the actual change.

Before jumping into the explanation, I'd like to mention some of the issues we faced in the past while finalising ZEP1 and ZEP2; they were:

• The current process allows the community members to propose a change to the specification (core/extension) and then put the ZEP to a vote without fleshing out the implementation details in various programming languages, which could be helpful for the implementation council. Though the author provides a reference implementation, currently, there's certainly no way we can ensure that a proposal is at least implemented/tested by ZIC maintainers other than the author. In the past, this led to issues like V3: node_type considered harmful zarr-specs#269, v3: dimension_names issues zarr-specs#268 and many more.
• During the ZEP2 voting timeline (which was three months), we received a lot of constructive feedback from the community members and the ZIC members, which helped improve the proposal. I realised the most vital feedback didn't arrive until we were close to the deadline. This raised a point - in our current process, the voting members and the community only thoroughly engage with the proposal when the voting deadline approaches. This is normal, as I understand that availability differs from person to person.
• After finalising ZEP1, many have wondered why we are allowed to change the specification after the voting has concluded. Please see Rename endian codec to bytes zarr-specs#263 and ZEP0001 status clarification, not "Final"? zarr-specs#265.
• The current process requires the creation of a new ZEP proposal, even if the change is minor and straightforward. This doesn't help and demotivates the author. Please see Add zstd codec zarr-specs#256.

A few solutions for the above-mentioned issues that I can think of:

• Having one or more reference implementations other than the author's in a different programming language before the voting and finalisation would be helpful.
• I think if we could dedicate a period where the voting members and the community read the proposal and provide comments and feedback before the actual implementation starts, we could finalise ZEPs efficiently and not in a manner where most of the critical feedback is pushed in the last stage at a single time.
• Adding a fast-track ZEP type, which allows the addition of minor changes without going through the whole process.

[MSanKeys963]

Based on my solutions mentioned above, here's the proposed revision for the process:

As mentioned in #55 (comment), a ZEP would progress in a phased manner. They are:

→ Reading Phase 📚

The Reading Phase, or Request for Comments (RFC) phase, initiates the ZEP proposal process, marking the introduction of the proposal to the Zarr community.

1. Stakeholder Engagement:

   • During this phase, stakeholders actively participate by expressing concerns and opinions related to the proposal.
   • The author establishes a GitHub issue to facilitate discussions, addressing questions, doubts, and concerns.

2. Duration Definition:

   • The author communicates the duration of the Reading Phase when submitting the ZEP, typically ranging from a suggested 3-4 weeks with flexibility for exceptional circumstances.

3. Feedback Collection:

   • Authors anticipate and welcome suggestions from readers to enhance the proposal, encouraging general comments to refine definitions, add URLs, or improve overall readability.

4. ZIC Acknowledgment:

   • ZIC members publicly acknowledge their engagement during the Reading Phase, ensuring a comprehensive understanding of community involvement.

5. VETO Mechanism:

   • A VETO option is available to ZIC members if the proposal introduces potential breaking changes to existing software implementations, serving as a precautionary measure.

→ Implementation Phase 🛠️

The Implementation Phase encompasses the timeline during which interested adopters actively work on integrating the proposal into their open-source codebases.

1. Declaration of Intent:

   • Before committing to implementation, ZIC members declare their intention to adopt the proposal, contributing to the formation of a cohesive adopter quorum.

2. Reference Implementation:

   • The author provides a reference implementation upon ZEP submission, a crucial resource during the implementation phase.
   • The inclusion of an Implementation Notes section in the ZEP proposal further assists ZIC members in understanding the implementation nuances.

3. Collaborative Iteration:

   • Implementors are expected to actively engage by raising issues, providing constructive suggestions, and highlighting areas for improvement in the ZEP proposal.

4. Feature Completion Table:

   • For proposals with divisible implementation details, maintaining a real-time Feature Completion Table ensures effective coordination among the Zarr community and stakeholders.

5. Concluding Implementation Phase:

   • Once a majority of implementations are completed within the established quorum, indicating the conclusion of the Implementation Phase, vetoes are no longer accepted.

6. Veto Transition:

   • After the completion of the Implementation Phase, vetoes are no longer accepted, establishing a clear point of transition and signalling the successful conclusion of the implementation process.

→ Voting Phase 🗳️

The Voting Phase represents the final stage where the ZIC and Zarr Steering Council (ZSC) formally vote on the ZEP proposal.

1. Notification:

   • ZIC members receive formal notification of the initiation of the Voting Phase.

2. Casting Votes:

   • During the vote, ZIC and ZSC members cast their votes as YES, ENDORSE, or ABSTAIN, providing clarity on their stance.

3. Vote Definitions:

   • YES indicates the member has implemented the ZEP and has no objection to the proposal.
   • ENDORSE signifies a commitment to future implementation, with no objections to the proposal.
   • ABSTAIN indicates a member's decision not to implement the ZEP but expresses no objection to its proposal.

4. Majority Approval:

   • The Voting Phase concludes upon achieving a majority of approval from voting members and resolving any pending comments or feedback.

5. Finalization:

   • After this phase, the ZEP proposal attains a FINAL status, and no further changes are permitted, ensuring a stable and conclusive outcome.

Here's a flowchart for a better visual understanding of the different phases:

[MSanKeys963]

I agree that my proposal is far from what @rabernat has proposed initially. I also agree that the current process makes it tedious by creating multiple issues in various places, which can lead to a loss of motivation and momentum. Combining repos could be of help here. But we should not eliminate the ZEP website as it helps keep all the proposed ZEP proposals organised, user-friendly, and readable. Also, I publish ZEP meeting notes for the community here: https://zarr.dev/zeps/meetings/.

I understand that the current goal is to simplify the ZEP process, and my proposal may add extra details/layers to the existing one. But I'd like us to consider the lessons we learned while finalising ZEP1 and ZEP2 while revising ZEP0.
I wonder if we can achieve a common ground between the simplification and my proposed proposal.

Please excuse me if I missed any details or something doesn't make sense. I'm happy to reiterate and correct based on the comments/feedback. This is the first attempt at revising the process, and I may have overlooked some important details.

Please let me know if you have any comments, questions, or feedback. Thanks!

[jbms]

I think we can have do everything in one PR as Ryan suggested, with the exception that if a discussion in a PR gets too long and complicated some parts of it could move to separate issues that are then ideally linked from the top PR description.

I also think that Sanket's proposed changes to emphasize implementation experience are also very helpful, and compatible with doing everything in a single PR. The couple changes I'd suggest to what Sanket has proposed, as we discussed today in the ZEP meeting:

• Indicate that quorum is 2 implementations with separate authors
• Merge final veto with final voting phase.
• Perhaps change "YES" vote to "YES and IMPLEMENTED" for clarity.

[clbarnes]

Minor thing: wherever the conversation(s) around a ZEP ends up, be it a PR or an issue or a github Discussion, it would be great to include a link to that/ those on the web page. Possibly that could go in the template?

[LDeakin]

Perhaps it is worth recognising that some ZEPs may never be "final". Some changes may be necessary post-acceptance (even if it is just fixing some ambiguous language, for example) and issues might not become apparent from just two implementations.

I am a fan of proper semantic versioning of ZEPs and their artefacts, with 1.0.0 on acceptance.

• This would make ZEP changes more discoverable. For example, the sharding codec added "index_location" in Adds index_location to sharding codec zarr-specs#280. That is noted in the sharding spec but omitted from the ZEP.
• The overall state of an implementation can be inferred by the ZEP versions it supports¹. This could be in a big table on the https://zarr.dev/implementations/ website, and this might spur more community effort to keep the various implementations across languages up-to-date.

Footnotes

1. And whether that support is full or partial. ↩