Formalize (and rethink) layering

[nightkr]

We started to touch upon this in #633, but I think it's worth breaking out into a separate issue.

Currently I think of the kube crates as essentially forming 5 layers:

1. Sans-IO request builder (kube::core::Request)
2. IO (kube::Client)
3. Typing (kube::Api)
   • At this layer we essentially have what the K8s team calls a basic client
4. Helpers for using the API correctly (for example: kube_runtime::watcher)
5. High-level abstractions for specific tasks (for example: kube_runtime::controller)

Currently most of our documentation pushes Api (layer 3) as the primary API, with layers 4 and 5 as exception cases that you more or less need to find on your own.

[nightkr]

Per the discussion in #633, it might be worth either making a new primary API or adapting kube::Api to fit better into layer 4 of this hierarchy, emphasizing how we want it to be used rather than the primitives that Kubernetes exposes (not that they're always bad per se, something like watcher simply cannot be done without a lot of client-side support).

[nightkr]

Neokubism (#594) is essentially about typing the sans-IO layer. That would free us up to start merging layer 3 and 4 while providing a reasonable way to bypass it as needed.

[clux]

I think in general this sounds like a good delineation.

Some specific questions on where we should draw the line (with my best guesses):

• 4: client-go like tools such as leader election / event recorders
• 3: portforward/remote-command: we integrate these more tightly against the api even though they are very special verbs
• 4: restart helpers and are not strictly Api based but conventions for certain types (probably put this in 3 wrongly)
• 3: discovery is a tangential api-specific concept (although has a bunch of conventions baked inside it for api resources)
• 3: dynamic api functionality (guess a lot of it is also traits in layer 1)

I guess thinking about it, most my questions are around the delineation between 3 and 4. and if you're considering merging the two then maybe this can be tackled more cleanly then, but there are a lot of stuff in in category 3 and 4.

If we get to where the neokubism approach is mergable; how do you propose we interface watch vs watchers in a merged 3 and 4 setup? That type of boundary is the hardest for me to visualise right now.

[clux]

Just saw a very similar pyramid at a talk at kubecon: https://youtu.be/uKF8v9X6hSE?t=423 but from a golang POV. Turns out the layering has a reasonable analogue.

0. go structs (k8s-pb/k8s-openapi)
1. apimachinery (core types)
2. apiserver (core Request)
3. kubectl (Api + Client)
4. controller-runtime (runtime)