Health of Linkerd project

[dims]

The TOC has determined the recently announced changes to Linkerd releases warrants discussion with the project concerning their health and this recent change. This issue is created to track the discussion and engagement with project and evaluate their current alignment with the expectations of a graduated project.

[monadic]

People and companies work on projects for economic and intrinsic motivations.

Eg

Economics of CNCF needs to justified to vendors.

Intrinsic motivations include belief that the project will flourish and be sustainable, popular, cool.

End users often are a mix of both. They will need support over time (in 3-10 year horizon) and in short term want to mitigate new tech risk

CNCF needs to take this a lot more seriously or it will be the home of poorly staffed second tier projects.

TOC+GB+EUC needs to engage at the level of product, marketing and long term business support, or find people who can.

[dims]

@shinebayar-g @monadic thanks for the comments, but let's please keep this issue focused for now. feel free to use other channels of communication (email thread, another issue, slack ..) for the larger debate.

[monadic]

admirable to address separately if possible.

unlikely to succeed because issues are tangled. linkerd governance seems to have fallen into disuse and obviously needs explaining & renewal. but real issue is why linkerd main developers felt obliged to adjust to an inferior OSS offering, which as far as I can see they are free to do, if they have to support it.

[kzap]

At our company, we have been using Linkerd in production for 4+ years. We have been thoroughly impressed by the level of maintenance on this project and work put into the project despite us paying $0 to the vendor Buoyant. It's quite understandable how this decision could come to fruition given the economic challenges that everyone in the industry including us are still going through. Things could have been far worse.

---

I have some suggestions for the CNCF:

From https://www.cncf.io/projects/

Graduated and incubating projects are considered stable and are used successfully in production environments.

Perhaps there can be some modifications to the Graduated project qualifications to encourage seeking more diversity in maintainers for a project to be listed as graduated.

The requirement of Committers from 2 organizations seems like a low bar given the reach and usage of Graduated projects. Making this atleast maintainers from 2 organizations (or having atleast 33% of maintainers from a different company) could be more helpful in using the Graduated status to incentivize having competing concerns at the maintainer level and lead to a healthier project. The CNCF and its community has grown so much since Linkerd graduated that it would only be natural to raise the bar.

If we look at Cilium maintainers we see a similar pattern of majority 1 vendor being the maintainers and thus could have commercial pressures to have those maintainers perform similar decisions to a graduated CNCF project like Cilium.

Also hope there could be some expectation of standards for releases and the naming of these versions. The edge vs stable is certainly confusing for end users of a graduated project.

This project has been well maintained by 1 vendor, perhaps too well. It is obvious that they have been wearing both open source and vendor hats at the same time so it is hard to discern if the maintainers are speaking with their open source hats or their vendor hats.

---

I am glad that this project is stewarded by the CNCF and all the trademarks and IP is owned by them. I am sure a positive outcome for the entire eco-system will come out of this, because thats how open source works and its why we put the time and effort into it whether we are end users or maintainers.

[lizrice]

There’s some healthy conversation to be had here about requirements and expectations, but I’m not sure it’s helpful to imply that Cilium is on the same path as Linkerd. I want to set the record straight before these misapprehensions take root! Yes, the majority of Cilium committers (the term Cilium uses for “maintainers” i.e. people with the commit bit) are from Isovalent, but we also have

• committers from 6 different companies plus another two non-affiliated committers
• a voting cap so that no company has a majority
• most importantly, an incredibly healthy flow of contributions from hundreds of contributors across dozens of organizations
• absolutely no plan to stop doing OSS stable builds!

It’s only a few months since the TOC gave Cilium’s governance a clean bill of health for graduation. Which isn’t to say that it’s perfect, and we’re actively working on improvements. At the moment we have 21% of GitHub contributions coming from non-Isovalent people, and we’d love to increase that number. I’ve had conversations in the last few weeks with folks from two additional companies, telling me they are assigning folks to working on Cilium full-time, in the hope of building up the expertise and trust that they can in time become committers too - this is great!

The reality is that projects need huge investments of time and resources, and that investment comes almost entirely from companies. Many foundation projects are majority supported by one company that has a commercial distribution, and is existentially bound to the success of the project. As a foundation community, we largely close our eyes to the financial health of those businesses, although it’s an incredibly important indicator of the investment into and therefore health of the underlying projects.

[steve-gray]

I'd like to raise five points, and I'm not really expecting answers to any of them:

• The change to what branches are made or binaries produced breaks no specific rule that anyone is able to articulate.

  • Does this mean there ought to be some? Maybe, but right here and now the rules aren't broken.
  • People don't like it, and you can argue semantics to try and find a violation-ish thing.

• The metrics being chosen here are poor, and people are selecting them to suit the discussion they want to have.

  • Cillium might have more numerical committers from other companies, but at the same time it's all about the measurement axis you want to use.
  • Being specifc, 42 out of 49 listed committers in MAINTAINERS.md are Isovalent, but on a pure lines of code and commits basis, its very much clear that the ratio is far more skewed than this selected metric indicates, with the ratio actually bordering on an effective totality.
  • Looking at commits by volume and originator, it is evident the withdrawl of Isovalent support would essentially have an impact as great as Buoyant's contribution to Linkerd, on a proportional basis.
  • The recent changes in ownership do put a valid question over the continuity of support for the open development of a product. A change in policy by Cisco would have a dramatic, likely cataclysmic impact on forward velocity.

• Members of the CNCF do not appear to be approaching this with clean hands, at least from an optics perspective.

  • Are there policies that indicate if you have a employment or other commercial interest that is geared towards a competing project, you recuse from deliberations involving those projects?
  • I would like to single out for example, Craig Box, as someone who is unbashedly partisan, with what appears to be a sustained grudge against the Linkerd project and its key people. But @lizrice too, obviously as an outsider to this process - on what basis can someone like myself feel your take is fair, genuine and not otherwise influenced by these relations?
  • It would stand to reason that recusal of people with such interests would still leave sufficient people for any deliberations on issues, given the deep bench the CNCF has to play with in terms of people?

• The reality is that the more deeply technical a project is, it lends itself to contributor concentration, as there generally is only ever one truly succesful commercialisation of a CNCF project.

  • This means that generally outside of projects favoured by one of the backbone technology companies, this pattern of concentration repeats itself.
  • The choice of licence and structure itself is explicitly intended to help vendors both contribute, yet commercialise - but beyond this ideological fork in the road, there is limited to no support. Many folks, for example Adam Jacob formerly of Chef have got a very nuanced take on this on X/Twitter, where he calls out the bind that CNCF often leaves companies in - but its echo'd by folks from WeaveWorks too (Edit: He's commented on this further up here too, I didnt put the github names together).

Food for thought.

[JorTurFer]

Honestly, I don't think that this issue is the place to talk about cilium or any other project, as this is for checking the health status of Linkerd as CNCF project. If there is any other project whose health can be discussed, let's do it in their own health checks

I've checked some of the commitments done during the graduation process and I'd just want to add my 2c providing some information for who is interested on it.

Have committers from at least two organizations

This is a requirement that all the graduated project have to address, in this case Linkerd was an exception, but they did some commitments:

Throughout its history, Linkerd has always been open to contributors and
maintainers from anywhere in the world. Linkerd has publicly committed to open
governance
and features a steering committee comprising end-users.

Checking the governance section, I can see that there is a maintainers section, where current maintainers are listed:

But Buoyant's CEO assumed the ownership of the decision in their own blog, despite he isn't present in the current maintainers panel. He is just who pays their salaries (he told that, not me):

There is another interesting point related with the steering committee. They (in theory) meet periodically not less than once a quarter. Recordings and minutes will be posted publicly., but checking the channel where those records are posted, there isn't any record since 2021. The last one happened only 2 days after graduation and never more. I think that this is interesting because one of the responsibilities is to "Provide neutral mediation for non-technical disputes." and this looks as a good point to be discussed there and records posted publicly.

Disclaimer: I'm not saying that they haven't met, I'm just saying that the public place where they shared the meetings haven't shared any other meeting after that and I can't find any other place where they are posted (but definitively it can exist and I just haven't found it)

[steve-gray]

Honestly, I don't think that this issue is the place to talk about cilium or any other project, as this is for checking the health status of Linkerd as CNCF project. If there is any other project whose health can be discussed, let's do it in their own health checks

If folks want to signal that somehow this is an issue here, and that "Project X" and "Project Y" they happen to be involved in is somehow good examplar, then I'm afraid that that does imply that a discussion of the comparative behaviours and realities of projects is fair game. It also lends itself to the question "Is this a Linkerd issue, or a wider problem with the framework under which projects operate in the CNCF?" - I feel ultimately the deck is rarely stacked for success.

I think any discussion of this issue would be remiss to not point out that a dramatic subset of the CNCF graduated projects have similar issues, and that list of 25 would whittle down pretty hard if people want to start drawing lines in the sand. I cited specific projects when they were brought out by those involved them directly, but it's obvious that you don't have to spin the spotlight around too far on its mount to find many, many other similar situations. Now maybe that is as important to discuss?

Have committers from at least two organizations

This is a requirement that all the graduated project have to address, in this case Linkerd was an exception

Is there a policy around reviewing the grants of these exceptions, a process for periodically renewing or revoking them? Going back and changing the deal years later, once the trademarks were signed over and after many years of hard service from the team behind the project is poor form.

But Buoyant's CEO assumed the ownership of the decision in their own blog, despite he isn't present in the current maintainers panel.

To break this down:

• He's able to put into effect whatever policy he wants around Buoyant though, and given their role in the maintenance of the project, he's as good a voice as any to be the figurehead on the discussion - I don't think it'd be fair or proper to just to grab one of the MAINTAINERS.md list and have them take the heat on this?
• I think this assumes that the maintainers didn't collectively agree to it, and that Will's not just putting a voice to it. If that's the case, is there a policy that these announcements have to come from someone in MAINTAINERS.md?
• The reality is that it's pretty much the norm people from the "vendor" side to be involved in public-facing statements, blog posts or media from the OSS projects without being in MAINTAINERS.md. I'm happy to provide citations of specific examples of this from other projects (it took minutes to find a few).
• Just like the general vendor concentration issue, this is an issue prevalent in CNCF graduates. I can see one particular example of a project where someone effectively speaks as the voice of the project, is in MAINTAINERS.md, but actually hasn't maintained the project from a technical level personally in years?

I'm actually not saying this is a problem either, but its far from unusual.

There is another interesting point related with the [steering committee]...........
.... I think that this is interesting because one of the responsibilities is to "Provide neutral mediation for non-technical disputes." and this looks as a good point to be discussed there and records posted publicly.

If the principal duty of that committee apart from "assisting" maintainers is in various ways is to discuss disputes, has anyone raised one on say, GitHub issues for the project, or somewhere else? I mention it because apart from one thread in Slack, nobodies actually bothered to formally raise a dispute with it. Going to be a quiet chat otherwise. The reality is that the decision has been met with quiet acceptance on aggregate, people don't like it fully, sure, but the reality is that the majority of discussion on the issue has been heavily brigaded in nature, taking place in non Linkerd forums.

(To qualify here, its not a Github issue on the project yet, nor has it had any discussion on Slack beyond one thread where a few folks who are patently not users of the project turned up to presumably controversy-farm for quotes in the public channels).

[JorTurFer]

I fully agree that this is something deeper than just Linkerd, but once again, this issue is to track Linkerd health.

I don't think it'd be fair or proper to just to grab one of the MAINTAINERS.md list and have them take the heat on this?

The election of the words is quite important for official communications, something like "the project", "maintainers", etc... could be more appropriated than the CEO of the company which will offer the enterprise license saying that it's his decision

To qualify here, its not a Github issue on the project yet, nor has it had any discussion on Slack beyond one thread where a few folks who are patently not users of the project turned up to presumably controversy-farm for quotes in the public channels

I personally asked in CNCF's Linkerd channel, as can be checked here 2 days ago, and I'm also a Linkerd user.

Honestly, I'm not trying to say that Buoyant is the evil, I fully understand their movement because they are a company trying to be sustainable. But it doesn't mean that I can't say that IMHO the project as CNCF project isn't doing the things well. There isn't any hidden roadmap behind my opinion at all, but as maintainer from another graduated project, I had to pass through a process that at least I thought that was to prevent these situations.

This coin has two sides, what will happen if all the projects from the CNCF landscape only offer stable artifacts behind a paywall? Will it be acceptable too? That's why I strongly suggest maintaining the issue's goal as other projects can be interested too on the result.

[steve-gray]

I fully agree that this is something deeper than just Linkerd, but once again, this issue is to track Linkerd health.

Understood. However I feel that if this discussion is taking place on what is.... very soft sand in terms of anything being done wrong, then it's pretty clearly going to need to morph into a "Do the rules need to change for projects being graduted" chat.

I personally asked in CNCF's Linkerd channel, as can be checked here 2 days ago, and I'm also a Linkerd user.

I stand corrected - in my defence had no idea that was an official project channel - and I suspect given the moderate activity in there generally nor did almost anyone, it's probably a forum that people pass an eye over.... irregularly. It's not even mentioned on Linkerd.io anywhere.

Honestly, I'm not trying to say that Buoyant is the evil, I fully understand their movement because they are a company trying to be sustainable. But it doesn't mean that I can't say that IMHO the project as CNCF project isn't doing the things well.

Thats a perfectly valid question. I think the issue is that 'the things' are generally poorly defined, and that general setup of things is such that there are many slow-moving fireballs moving roughly in the same direction amongst CNCF projects.

There isn't any hidden roadmap behind my opinion at all, but as maintainer from another graduated project, I had to pass through a process that at least I thought that was to prevent these situations.

I wasn't referring to you on anything in particular on this front - but I simply meant that people with a vested commercial or personal interest in Linkerd getting a bruise being part of these discussions feels a little warped, especially when wheeling out their own projects to justify their criticisms. You'd be a good example of someone without a fighter in the ring on this issue.

This coin has two sides, what will happen if all the projects from the CNCF landscape only offer stable artifacts behind a paywall? Will it be acceptable too? That's why I strongly suggest maintaining the issue's goal as other projects can be interested too on the result.

Its an interesting one, and one that probably requires some decisions about how to come up with a more prescriptive policy around this that outlines the expectations and requirements. I mean the requirement totality for a graduated project is A clear versioning scheme.. Now I do feel two things are true concurrently:

• The scheme they're moving to is clear
• Its controversial.

A clearer, more prescriptive standard on what's expected here wouldn't hurt the CNCF to have - but applying it to all projects fairly, backporting it into their processes etc will be interesting - especially once those projects with a single key vendor start seeing the scope of mandatory process increasing.

[christianhuening]

I read @kzap 's comment to be less about Cilium, but rather about the pattern and that this situation could relatively easily happen to more projects not by ill intention but mere chance and the dynamics of the cloud-native (business) world.

If Isovalent would for some reason have to make the decision to stop investing time into stable OSS builds, the responsibility to do so would rely on the remaining non-Isovalent committers/maintainers.

What I am trying to say here, is that the decision to stop providing builds tagged as "stable", has been done by the company currently investing the majority of maintainers to the project LinkerD. So if there were committers from other companies, that would make for a more diverse (healthier?) ecosystem but would also not guarantee the publishing of stable, quality checked release builds in and as of itself.

So the discussion should not be about builds or no builds, but rather about what we do with projects that fall below the required minimum (for maintainer diversity) and whether the minimum requirements contain the right metrics to look at.

[monadic]

Hi, if this is important why don't some other people step in and provide builds?

This is preferable to any of us telling the buoyant people what to do or not.

[kingdonb]

It sounds like it will be complicated for a third party to provide the stable builds, unless I've misunderstood something; there will not be stable tags published anymore in the open source repo, so for a neutral third party to provide the stable builds is not going to be a trivial exercise. https://twitter.com/wm/status/1761187286153064578 It either wouldn't be published as a tag, or it would be published in a repo with the BEL applied. It's not exactly clear to me which of these it is (I think the latter)

Maybe it would be a trivial exercise after all re-publishing those build artifacts, but it's worth asking if someone did this and it became the most popular distribution of linkerd, would that be legal to distribute in circumvention of the BEL terms, and if it was or wasn't, then wouldn't doing that undermine the goals of Buoyant (who are still doing the lion's share of work here)

this change makes space for Buoyant to build a sustainable business around Linkerd

From the clarification blog on buoyant.io: https://buoyant.io/blog/clarifications-on-linkerd-2-15-stable-announcement

I'm not saying that edge releases or stable releases are or aren't more important for the people that linkerd has intentionally exempted from the BEL, and we can test the boundary, but I don't think it's going to be helpful to undermine the clearly stated goals of the project team.

[campbel]

so for a neutral third party to provide the stable builds is not going to be a trivial exercise.

Creating builds with tags is the trivial part. The non-trivial part is the qualification process and keeping out partially completed features among other things. My understanding (maybe from one of Williams posts) is that the process of creating these builds is time consuming, but not the reason for the change.

...if someone did this and it became the most popular distribution of linkerd, would that be legal to distribute in circumvention of the BEL terms

BEL terms don't apply to Linkerd OS, which is apache 2 licensed. Anyone could start selling stable releases of Linkerd tomorrow, just as Buoyant is, they are no more privileged to do so.

I'm not saying that edge releases or stable releases are or aren't more important for the people that linkerd has intentionally exempted from the BEL, and we can test the boundary, but I don't think it's going to be helpful to undermine the clearly stated goals of the project team.

This is the crux of the issue. Any effort to subvert the new system is going to be at odds with the long-term health of the project as Linkerd is existentially tied to Buoyant.

[mikebell90]

Full Disclosure: I evaluated them a couple of years ago, and really liked their support, design, and interaction with engineers. But I went with Istio with great regret, because they were missing some needed features.

I agree with some sort of way to keep Buoyant, er buoyant ; ). People that don't contribute money or code make it impossible to maintain a for profit company. And ironically one reason people don't contribute is because it's very stable and very easy to install. (which ironically hurts them here - we had to purchase a for profit distribution of Istio to get things working the way we want)

However

1. The timeframe is problematic for many organizations. Many commit budgets yearly. I suppose that's Buoyant's (in)decision but it's pretty iffy - had they announced this in November, with the calendar year coming, no problem.
2. It's a little weird to sell two tiers of sales - one with no support at all, just access to the builds, and one with support. I guess that can be defended, but....
3. It seems like all they are doing is separately forking stuff to an internal repo, and doing the packaging there. If the tags are clearly marked and available on their public repo, and all they are saying is they won't do the builds, fine. I don't think that's what happening. I guess they will cherry pick, otherwise you could just trace the git sha.

Much of this is addressed in https://buoyant.io/blog/clarifications-on-linkerd-2-15-stable-announcement which is worth reading.

Moving to the more relevant part for CNCF

1. This is still open source. It's annoying, but it's open source.
2. Maintainers - here I do think CNCF should more carefully scrutinize the diversity of the committer ecosystem. Otherwise companies are just using CNCF for bragging rights.
3. Steering Committee - I mean CNCF can do what it wants, but I'd like to argue a "normal" (subjective) steering committee meets regularly (at least quarterly), and normally has influence over many aspects

AND HEY https://github.com/linkerd/linkerd2/blob/main/STEERING.md their charter says exactly that and shows that they are unfortunately possibly in violation of their own charter.

The Steering Committee’s responsibilities are to:

Provide feedback to project maintainers about Linkerd's featureset, UX, and operational model.
Assist Linkerd maintainers in prioritizing upcoming roadmap items and planned work.
Represent the "voice of the Linkerd user" both internally and to the broader community.
Provide neutral mediation for non-technical disputes.
Develop and maintain a project continuity plan.

Meetings
Meetings will happen periodically not less than once a quarter. Recordings and minutes will be posted publicly.

Changes to this charter
Changes to this charter must be approved by a majority of Steering Committee members and a majority of Linkerd maintainers

[wmorgan]

Hi all. Just wanted to say that the Linkerd maintainers and I are following along with the discussion here but we probably won't be chiming in more substantially until it's a little more clear what is being asked of us by the TOC.

Linkerd has been an integral part of the Kubernetes and CNCF ecosystem since its donation over 7 years ago and I believe we all have the same goal: the long-term survival and growth of the project and the ecosystem around it.

I am confident we can make this discussion a collaborative one ("how can we make this work for everyone, both maintainers to users?") rather than a punitive one. The eyes of future potential CNCF projects are upon us now—and many of them will look exactly like Linkerd does.

[fernandoescolar]

I think this thread is not about whether Buoyant is doing things right or wrong. Anyone can think both.

This discussion is about if Linkerd should still be part of CNCF. As far as I know, it seems they have violated some terms of CNCF, so it should be evaluated.

A lot of people rely on CNCF graduate projects and it's not just about open source.

[kingdonb]

To riff on that further, I don't think the "new product lineup" or the BEL additions are either of them necessarily a violation in question, the only substantial violation I've heard described so far in the thread is the failure to have a public meeting of the oversight committee on a regular basis, where feedback can be offered and a recording of what was said can be kept for posterity. A remedy would be to meet and follow the charter once again.

If the terms of the new license aren't aligned with the CNCF rules then maybe a second violation of the community branding rules - can you still call it linkerd if it's not a product aligned with the rules of the CNCF, or do you need to call it something that makes clearer the separation from the CNCF graduated project that has its own trademark policy and use restrictions on the name. A remedy would be an adjustment of the use of marks. (Anyone at all is free to take the Apache 2 sources of linkerd, make it into a saleable product with their own additions, and sell it with or without source available, according to the terms of the license afaik - they only need to abide by the trademark policies.)

Just trying to help ff to the point where it is clear what we are asking linkerd maintainers to do for project health, and for alignment with the rules of the foundation. Thanks for helping to focus the discussion.

[craigbox]

Hello! Thank you for summoning me. I apologise in advance for derailing the thread: I was not going to chime in either, but I'm being subtweeted and impugned enough that I think it worth clearing up a few points.

Members of the CNCF do not appear to be approaching this with clean hands, at least from an optics perspective.

The CNCF is a subproject of a 501(c)(6) nonprofit mutual benefit corporation. Its members are companies¹. Platinum members are given a seat on the Governing Board, gold and silver members get to elect three each. Projects, and their maintainers and developers, are entitled to exactly two seats at this table.

Liz Rice and I are both on the Governing Board; her as an elected representative of the Silver Members, me as an elected representative of the projects.

I have one vote out of 30 members on anything that comes before the GB². I expect at some point I'll be asked to approve the project's budget.

cncf.io states "The GB does not make technical decisions for the CNCF other than working with the TOC to set the overall scope for the CNCF."

Neither Liz nor I are on the TOC. Even if we were, we would still be allowed to have opinions, and we would still be allowed to put them forward for discussion.

---

I would like to single out for example, Craig Box, as someone who is unbashedly partisan, with what appears to be a sustained grudge against the Linkerd project and its key people

I've been affiliated with Istio since 2017 and had a leadership position in the project since 2020.

I remember when the projects were "peanut butter and jelly". And then they weren't.

If you're looking for a sustained grudge, the "key people" at Buoyant have, in my opinion, been openly antagonistic towards Istio since that point. The examples are easy enough to find.

From my side, you're welcome to check out how I showcased Linkerd on the Kubernetes Podcast, or talk to the Linkerd community members I collaborated with while leading the PC for ServiceMeshCon. With my collaborative hat on, neutrality and fairness are important to me.

I have been active in the last few months in advocating for changes to how the CNCF handles projects; some context here. I stood for my GB seat on this platform. My interest in highlighting this issue is as much about having a level playing field in the CNCF as it is about any one project.

This is not a new discussion. I think this issue bears reviving.

I don't speak for the CNCF while tweeting; not even 1/30th of it. I don't believe anything I have said was untrue, and I'm happy to post corrections if it was.

---

I'd like to raise five points, and I'm not really expecting answers to any of them:

That was only four. So, let me borrow a fifth point from a post of yours on Reddit that someone kindly forwarded to me:

I think one of the interesting things that I've learned in the last 24 hours is that the one person who sits on the CNCF with lots to say about service meshes happens to have a vested commercial interest in a Buoyant competitor, historical affiliation with alternatives to Linkerd, and a very cute-but-petty Twitter flame-war going on against Linkerd and Will. The behaviour of brigading people online to proxy your own arguments wouldn't pass muster on a subreddit, but apparently it's alright if you used to work for Google? o_O

• I'm currently employed by Solo, but they are included in the list of groups I am not speaking for. (And my stock in them is unvested.)
• My affiliation with alternatives with Linkerd is very current!

And, regarding the direct accusation of "brigading" as well as the indirect accusation of the same in William's last blog post:

• I haven't engaged in any brigading, I don't have or use any sock puppets (real or virtual), I haven't asked anyone else to participate in this debate, and I am being completely honest when I say I wish Buoyant well.

Footnotes

1. Or at the least, organizations; there are non-profit and academic members also. ↩

2. In the unlikely event that anything specific to this issue would come in front of the GB, be aware its members include "trillion-dollar behemoths like Google and Microsoft". ↩

[nemobis]

Hi, if this is important why don't some other people step in and provide builds?

1. "Political"/social reasons: it would presumably need at least a light fork (unless the linkerd project is willing to host the required branches, tags, cherry picking etc. on their main repository). Some governance questions would arise.
2. Resourcing: there would be at least some redundant effort. Does CNCF have means to gather financial contributions for shared services like packaging?

(This last question may become more relevant in the near future, for example if CNCF as OSSS/open-source software steward under the CRA were to become responsible of a minimum standard of security best practices.)

[mikebell90]

1. Political"/social reasons: it would presumably need at least a light fork (unless the linkerd project is willing to host the required branches, tags, cherry picking etc. on their main repository). Some governance questions would arise.

I mean if the release tags were already on the repo, the issue would be greatly trivialized (just a build farm). But my understanding is they don't intend to do this.

[mattfarina]

The Governance is point to look at for Linkerd. Here is what I'm seeing that raises a red flag to evaluate.

• The maintainers are all from Bouyant which does not pass the graduation criteria which requires committers from at least 2 organizations. Note, when you have maintainers from multiple orgs it forces you to keep project things outside the company to support everyone. It's a useful forcing function.
• The steering committee isn't one like Kubernetes with authority. See the steering governance. It appears to be a voice of the user group, instead. This means it doesn't solve for the multi-vendor org situation.

It doesn't look like the governance meets the letter or the spirit of the CNCF graduation criteria. This needs to be evaluated, IMHO.

[dlorenc]

It doesn't look like the governance meets the letter or the spirit of the CNCF graduation criteria. This needs to be evaluated, IMHO.

+1 to this. The one other point I'd suggest looking at is the release tag situation. I agree there's no need to provide release builds, but I don't know how the Linkerd Project can claim to have a 2.15 release if there is no corresponding release tag on the repository.

The project could opt to not have stable releases at all, but having the Linkerd project advertise the existence of a release that isn't even available in source form without using a (single) vendor is problematic.

[angellk]

Would like to highlight this blog post about Linkerd:

https://bitergia.com/blog/bitergia-analytics/decoding-linkerds-journey-in-the-cncf-a-dive-into-project-health-metrics/

[stefanprodan]

I find this discussion relevant to the future of all graduated projects who are maintained by a small group of people, regardless of being single-vendor or not. Not every project has dedicated people available to do release management, CVE back porting, multi-platform distributions, etc. Some projects could opt-out of the release management tasks and focus on other areas, if this is an acceptable practice to not ship stable releases.

I personally would like to draw some practical conclusions from this discussion:

• Is it Ok for a graduated project to not do CVE back porting to the previous minor? (e.g. Helm doesn't)
• Is it Ok for a graduated project to stop shipping artifacts for stable releases? (e.g. no container images nor binaries)
• Is it Ok for a graduated project to stop tagging stable releases? ( e.g. no vX.Y.Z Git tags)
• Is it Ok for a graduated project to advertise on the the project website the vendors and their commercial offerings of stable releases if these are no longer available upstream?

[monadic]

Ultimately projects should do what is best for those projects.

Now: We want projects to be sustainable and we want CNCF overall to provide an excellent tool-chain for Cloud native apps and infra. But to achieve those two goals it is important that users cam expect sustainable high quality from a project. And high quality projects do include usable stable builds, in my view. And maybe more in some cases. (it is not always feasible to have certain builds, f.ex).

The best way to have high quality sustainable projects, is to make the business model easier. Already it is hard enough to sustain any software company. If that company offers an OSS product then it trades off a free developer entry point for revenue at scale. That balance can be really hard to get right -- Gitlab are doing a nice job at the moment, but they control the end to end story and are a single-vendor product.

CNCF says that projects (and by implication, vendors) can succeed by loosening control of that end to end story (or "journey") so that all entry points can be in the community where (eg) vendors can fairly easily compete to create a better paid end products. Single vendors can work if they are able to manage that loss of control. Or multilple vendors can work well in some cases too. But it is harder than being Gitlab.

So I say we need to make it easier for ALL vendors here. Competition breeds fairness.

[dzolotusky]

The TOC met today in a closed meeting with a Linkerd maintainer and William Morgan (CEO of Buoyant and author of the 2.15 release blog posts) to discuss the challenges the project is currently experiencing, the expectations around graduated projects and how the CNCF can better support this and other graduated projects as they mature and move forward.

After an engaging and collaborative discussion with Linkerd, the TOC recommends the following next steps:

• We invite the Linkerd project to work with TAG Contributor Strategy as a next step. This group is the most likely to be able to identify areas of growth and strategies for the future. We suggest that the project file a Governance review issue with TAG Contributor Strategy to perform a comprehensive governance review of the project to ensure that the project's own governance and processes (which are specific to Linkerd) best set it up for success, adherence to documented governance, and provide appropriate recommendations for the project to implement.
  • We're expecting that the TAG CS review will work with Linkerd to confirm that the governance process is open and transparent; that it enables the project direction to be set by the Linkerd community in accordance with their governance, and that it does not openly favor one vendor or organization over another in setting its direction and therefore avoids introducing structures that permit favoritism to occur within the project. This is separate from the previous criteria of maintainers from at least 2 organizations which the project previously received an exception for and should also be evaluated for impact on the project as part of the review.
  • It is further requested that the project seek specific feedback from TAG CS on how to create feedback loops with both contributors and adopters to improve and understand needs and features being requested of the project as it pertains to their defined scope.
• Separately from the governance review, the Linkerd Project plans to update their proposed development roadmap to better define a “core” and “sub-project integration”. This already ongoing work is important as the project matures, and increasing scope, especially around integrations, has been flagged as a concern by maintainers.
• The project establishes appropriate structure and subsequent governance that expectations of Linkerd Core as well as sub-projects that provide integration and ease of use for adopters of Linkerd Core but are otherwise out of scope of the main project.
• Establish a named role to better define the position that’s currently being held by William Morgan but might be a different named individual in the future.
• The project establishes a roadmap that lays out when these activities are expected to occur and linked to this issue with an anticipated 6 month asynchronous check in with the TOC regarding progress against these milestones, and any further help that the TOC can provide.

Expected update: early September for completion on these tasks (6 months from now)

[dzolotusky]

For all individuals who have provided their insights, opinions, and observations on this issue:

The TOC opens “Health of $PROJECT” issues to understand the current state of a given project so we may engage the project, learn, understand, and provide recommendations to remediate concerns brought up. Community members are welcome to share their observations about the project’s health on the issue.

For discussions outside of the immediate health of the project in question, please leverage the Discussions area available on our repo to engage in meaningful, insightful, and constructive exploration of a given topic more generally. Please also bear in mind that participation in discussions should be mindful and considerate of the audience — your peers, maintainers, contributors, and community members. We expect these discussions to remain constructive and improvement oriented.

We understand our community is passionate about these topics as the outcomes of these discussions and the TOC’s decision on them impact many individuals, organizations, and projects. It is critical that our community weigh whether their perspective is a force of progress that enables positive, meaningful outcomes before commenting.

[wmorgan]

Thank you @dzolotusky and the rest of the TOC for a productive discussion today. I look forward to our conversations with TAG CS and I am confident we will confirm Linkerd's long-established commitment to open governance.

From my perspective many of the comments here have strayed from the collaborative, supportive environment that I believe the CNCF community to be. I urge those of you who have expressed strong opinions about Linkerd to come get involved in the project. We have an incredibly exciting (and daunting!) roadmap ahead of us and would love to welcome you aboard.

[monadic]

Fantastic work everyone

@dzolotusky URL for spin-off thread in Discussions?

[TheFoxAtWork]

@monadic All discussions are here: https://github.com/cncf/toc/discussions there is at least one started on the two org requirement

[monadic]

many thanks @TheFoxAtWork -- I hadn't joined the dots on what was what!

[dzolotusky]

@wmorgan do you have any updates on the items from our conversation back in March that are listed in my comment from March 12 (#1262 (comment))?

[wmorgan]

Hi @dzolotusky. Here's where we are:

• We've filed [Governance Review]: Linkerd tag-contributor-strategy#648 to request governance review from TAG CS.
• We've merged update governance to introduce a director role linkerd/linkerd2#12262 to assign me a more formal role in the project.
• In terms of subproject splitting, for the time being we're just biting the bullet here and keeping them under the same governance. (E.g. Add viz stat-inbound and viz stat-outbound commands linkerd/linkerd2#12994 is an example of recent maintenance to the viz extension.) I think we're going to continue evaluating this as we evolve, but the burden hasn't been as much as we've thought.

A couple other relevant items not explicitly asked for here but that have come out of conversations with the community:

• When we announce major versions, we're now tagging commits (e.g. version-2.15, version-2.16) and calling out the corresponding edge release artifact.
• To make edge release artifacts easier to consume, we're now publishing guidance in the form of post-hoc recommended / not recommended badges in the Linkerd GitHub releases page and also through monthly roundup blog posts (example).
• We've revived the steering committee. We met in May and are due for another meet this quarter.

[dzolotusky]

We had a quick discussion about this in today's TOC meeting and agree that we'd happy with the progress by Linkerd. I'll leave this open for another week for anyone else to comment. If there is no further comment, this issue will be closed.

[mrbobbytables]

There was a follow-up comment from @wmorgan in the TOC meeting issue: #1425 (comment) that I just wanted to drop here for posterity^^;;

If there's anything I can provide to help characterize Linkerd's health, please let me know. A couple datapoints come to mind:

• Since February we've shipped 30 edge releases (one a week!), improved our user-facing guidance for edge releases, and seen some great examples of the power of edge releases for rapid bugfixing (e.g. Intermittent routing failures with HTTPRoute  linkerd/linkerd2#12610)
• We've shipped many features that had been languishing in the backlog, including IPv6 support, an authz policy audit mode, and a new implementation of retries, timeouts, and per-route metrics—some of the earliest code ever shipped in Linkerd—that fixes some long-standing corner case behavior in these features.
• The Linkerd community Slack hit 10,000 members 🎉
• There are 9 talks at Kubecon SLC with "Linkerd" in the title or description 🎉

Lots more to talk about but I am writing a retrospective blog post and don't want to spoil too much :)