Formalize the concept of subprojects

[spiffxp]

This is based on / inspired by a SIG Architecture proposal to expand SIG metadata and flesh out OWNERS

This is driven by a steering committee goal of ensuring that our organizational structure maps to our code structure.

Every part of the Kubernetes code and documentation should be owned by a Special Interest Group (SIG). However, many SIGs have multiple significant subprojects with distinct (though sometimes overlapping) sets of contributors and owners, who act as subproject’s technical leaders: responsible for vision and direction and overall design, choose/approve change proposal (KEP) approvers, field technical escalations, etc. We should formally recognize that.

For example, SIG Network potentially could be divided into multiple subprojects: pod networking (CNI, etc.), Service, Ingress, DNS, and Network policy. SIG Apps has the workload APIs, Helm, and Kompose. SIG Cluster Lifecycle has kubeadm, kops, kubespray, minikube, etc. SIG API machinery has apiserver, client libraries, API discovery and OpenAPI, CRD, API aggregation, and admission extension. In addition to Kubelet proper, SIG Node has multiple CRI implementations underway. Maybe some of these subprojects should be grouped, but hopefully you get the idea.

I'd like to start by implementing this concept in sigs.yaml. This ensures the information is machine-readable, generated README's contain human-readable information, and we start with a single source of truth.

Strawman:

• a sig may have N subprojects
• a subproject must have a name
• a subproject must have 1 - N OWNERS files associated with it
• the OWNERS files define the root of the repos/subdirectories owned by the subproject
• a subproject may have a description
• a subproject may have one or more meetings
• meetings can have different uris for video, agenda, recordings, etc.

Looking at SIG Apps as an example:

• A "Workloads API" subproject could potentially span multiple subdirs and repos
• A "Charts" subproject spans the kubernetes/charts repo
• A "Helm" subproject could potentially span kubernetes/helm and kubernetes-helm/* repos

Looking at SIG Contributor Experience as an example:

• A "Mentorship" subproject would own the docs in kubernetes/community that related to mentorship
• A "Community Management" subproject would own the docs in multiple subdirs of kubernetes/community

I am putting together an implementation based on this strawman with a total guess of single-repo subprojects based on first-pass by @bgrant0607, so we have a concrete example to iterate on. The intent isn't to voluntold SIG's into owning repos, but to identify where our gaps lie.

/sig contributor-experience
/sig apps
Since I mention you two as examples

/committee steering
/priority important-soon
/assign

[spiffxp]

/sig architecture

[bgrant0607]

One idea that just occur to me is that maybe the area/ labels should be replaced by subproject/ labels.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[cblecker]

/remove-lifecycle stale

[grodrigues3]

@spiffxp where are we with this? some sigs do have sub-projects . Do we need to push the idea more broadly? Do we have any metrics for how well this has been rolled out/accepted?

[spiffxp]

There is little to no automation that consumes sigs.yaml outside of the docs generator in this repo. As a thing that is commonly understood, I still think we have work to do on sigs/wgs/subprojects.

To tighten things down I've got two umbrella issues:

• Umbrella issue: ensure all repos present in sigs.yaml community#2464 - get all repos in sigs.yaml
• Add OWNERS files to kubernetes repos that lack them community#1721 - add OWNERS files to repos that lack them

TBH right now I'm less concerned about the fact that we also haven't:

• added owners a field to OWNERS files (I have historically been treating approvers at root OWNERS as owners)
• setup any automation to audit this information
• tied subprojects to github labels in any cohesive way (anecdotally I try to use an area/foo label for subproject foo)
• setup any automation to auto-label issues based on the subproject they're opened against
• setup any automation to auto-label PR's based on which repos or subtrees they touch

I would say if we get to the point where we've got good label hygiene based on these we can track metrics

[nikhita]

There is a PR open to add a project-manager-like role for subprojects here: kubernetes/community#3413.

I think we can take a look at updating automation once we finalize on the subproject roles.

[timothysc]

poke, can we close this now?

[spiffxp]

Here's what's missing: a very tight machine readable definition of subproject owners. The sig-governance definition of the role is pretty crisp, except that it says they're defined in sigs.yaml.

Today in sigs.yaml, subprojects are defined by a list of OWNERS files, and those OWNERS files have "approvers". Are subproject owners the union of all of the approvers listed therein? The intersection of those approvers?

This is the problem I was raising with SIG Cloud Provider's impending merge of other cloud provider SIG subprojects. Ideally there would be one subproject per cloud, each composed of many repos... but then how do we denote the people who have final say for cloud foo, while also denoting the "tech leads" for each repo? Subprojects as a list of owners files lack the hierarchy needed to make this distinction.

Options to overcome this:

• A: Decentralize subproject owner definition in OWNERS files, pick either "union" or "intersection" of approvers in OWNERS files (ref: Subprojects in sigs.yaml will be instantly out of date community#1913)
  • Something would have to ensure sigs.yaml or some centralized definition is kept up to date
  • This feels like it would be prone to typos or "forgot to add to one last OWNERS file" style problems
• B: Centralize subproject owner definition in sigs.yaml, the same way that we currently do for SIG/WG/UG/Committee leadership (eg: Extend subproject and repo generation for generator app community#2731)
  • Something would have to ensure subproject owners have approve authority in all relevant OWNERS files
• C: Centralize subproject owner definition in GitHub teams, add support for teams in OWNERS files (ref: [prow] Add support for github teams in OWNERS files test-infra#10065)
  • Something would have to ensure the aliases have approve authority in all relevant OWNERS files
• E: Add a new owners: field to OWNERS files to denote people that are subproject owners
  • Unclear whether our automation would then need to consume it or treat them differently
  • Something would have to ensure the same owners are used across all OWNERS files within a subproject
• F: Mandate some form of OWNERS file hierarchy, and the top level OWNERS file wins
  • This doesn't conceptually make sense for subprojects that are spread across multiple repos (kubernetes-csi)
  • We could say that order of the list of OWNERS files matter, and the top one wins
  • We could make a new field for top-level owners, and then a field for all the rest of the owners files

None of these individually solves the tension between a desire to centralize the info, but distribute the authority. They all require something to walk OWNERS files and reconcile against or into a central place, so I'm going to start there.

[spiffxp]

Analysis of common owners https://gist.github.com/spiffxp/629ab537c2e1d59072aa3154c91459c0

[bgrant0607]

/remove-sig arch

[k8s-ci-robot]

@bgrant0607: Those labels are not set on the issue: sig/arch

In response to this:

/remove-sig arch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[bgrant0607]

/remove-sig apps

[spiffxp]

/remove-sig architecture

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[spiffxp]

/remove-lifecycle stale
/lifecycle frozen