Switch to Istio native sidecars

[mjnagel]

Is your feature request related to a problem? Please describe.

Currently we use istio's sidecars across all pods in the mesh. Due to existing istio issues we have also implemented a Pepr watch action to handle properly terminating sidecars for jobs. If we switch to "native" sidecars we will no longer need this action and may have some other benefits with regards to init containers accessing the network (common with things that validate dependencies that may be in cluster).

Describe the solution you'd like

Istio should be deployed with native sidecars by default (ENABLE_NATIVE_SIDECARS env on pilot). When upgrading this should be a seamless rollout - unsure if we will need to cycle all pods or something similar here (may be worth evaluating this issue first).

Definition of done:

• Native sidecars are enabled by default in uds core
• Sidecar termination watch is removed from pepr code
• On an upgrade all pods are properly migrated to native sidecars
• Documentation may need an update to ensure our k8s version pre-req is captured (this would require 1.29, i.e. non-EOL versions)

Describe alternatives you've considered

We could continue to use non-native sidecars + our pepr capability.

Additional context

This functionality in kubernetes is available in 1.28 behind a feature flag, and 1.29 by default. We should be fairly safe to enable this but should document this requirement well (maybe in our k8s/distro support page). In theory someone could use overrides to switch to non-native sidecars but they would be missing our pepr capability unless we maintain that behind a flag as well.

[adam-defenseunicorns]

Slack Thread Request - 9 July @Racer159

[mjnagel]

On an upgrade all pods are properly migrated to native sidecars

This will mostly be a change to our injection/killPods logic. In particular we will want to make sure we are filtering for the istio container as an initContainer now (here). We will also likely need something extra to modify the logic around when we kill pods (here), such as an annotation on the namespace indicating whether it has been migrated to native sidecars or not.