Provisional proposal to add API options to control where build pods are
scheduled. These will rely on core Kubernetes features related to pod
scheduling.

This proposal was partially motivated by the multi-arch feature
discussions, where it was revealed that we currently have no means of
controlling where build pods are scheduled. While these features may
support a future proof of concept for multi-arch builds, orchestrating
multi-arch builds end to end is out of scope.

- Removed the goal to set pod scheduling options at the cluster level.
  This adds significant scope since the build controller would need to
  support this configuration via a file or large set of environment
  variables. This goal is not ruled out completely - I'm recommending
  it as a follow up feature so the community can iterate and deliver
  quickly.
- Added support for custom schedulers - this feature has been in
  Kubernetes for quite some time, though its adoption is somewhat
  limited due to the challenges of writing a custom scheduler.

Signed-off-by: Adam Kaplan <[email protected]>

Affinity and anti-affinity is an incredibly rich and complex API in
Kubernetes. Most of the other scheduler tuning APIs in this SHIP are
simpler to understand and can be wired directly to underlying APIs
(Tekton today, potentially another implementation in the future).
Exposing the entire affinity rule API will make it challenging for
developers to utilize.

We should instead strive for a simpler API that makes it easy for
developers to express their intent. This abstraction can then be
translated to the underlying scheduler APIs. Such an effort is not
trivial and warrants a separate feature/SHIP.

Signed-off-by: Adam Kaplan <[email protected]>

These changes complete all sections of the SHIP needed to reach the
"Implementable" state. This included some cleanup in the top summary,
concrete implementations of new API fields with their respective
types, and justification for implementing our own Tolerations API.

As part of thinking through risks, mitigations, and drawbacks, I
recommend that the Shipwright community publish a hardening guide
alongside this feature's implementation. Exposing node selectors and
tolerating taints can risk system availability, especially if builds
are scheduled on control plane nodes. Kubernetes tries to prevent this
by default through node labels and taints. There are some scenarios
(ex: single node or small clusters) where scheduling builds on the
control plane is desirable or necessary; publishing a hardening guide
preserves flexibility while enabling informed choices for security.

The drawbacks and alternatives also call out potential future concerns
related to multi-arch support in Shipwright. This is under active
discussion, and SHIP-0039 may add additional complexity to any future
design.

Signed-off-by: Adam Kaplan <[email protected]>