Add ztunnel/cni components to Istio

[mjnagel]

Related to #681

In order to incrementally work towards ambient mesh we should add the ztunnel and cni components to our istio setup. Ideally these should all be on v1.24.0 (when ambient was GA-ed). Definition of done here:

• Ztunnel and install-cni components included in default istio setup
• Automatic detection or documentation on any distro specific configuration for the CNI setup (example) - if distro specifics are k3d exclusively this could be handled with bundle overrides?
• Full functionality of these two components in all 3 flavors (and FIPS images for registry1/unicorn flavors)

The scope of this issue does not require any workloads to be switched to ambient, or any support of ambient in the operator - but if those changes are necessary/helpful to validate ztunnel functionality they can be worked here.

[mjnagel]

#699 includes a draft of this work, although not all images are the correct versions or flavors at this point.

[mjnagel]

This may require working #1027

[sgettys]

Got the base ztunnel/cni components wired in and tested locally, going to do the same tests for injection and ambient packages on k3s, rke2, and eks to make sure we're not breaking anything

[sgettys]

Ran into this bug when testing out Istio CNI on k3s:
istio/istio#53862
This keeps Istio CNI from being able to setup the iptables for routing request:

│ 2024-12-12T00:16:03.424736Z    info    cni-agent    adding pod to the mesh    ns=pepr-system name=pepr-uds-core-75b945f5fd-6jdj2                                                                                                                                                                                                  │
│ 2024-12-12T00:16:03.425240Z    info    cni-agent    Running iptables-nft-restore with the following input:                                                                                                                                                                                                                        │
│ * mangle                                                                                                                                                                                                                                                                                                                          │
│ -N ISTIO_PRERT                                                                                                                                                                                                                                                                                                                    │
│ -N ISTIO_OUTPUT                                                                                                                                                                                                                                                                                                                   │
│ -A PREROUTING -j ISTIO_PRERT                                                                                                                                                                                                                                                                                                      │
│ -A OUTPUT -j ISTIO_OUTPUT                                                                                                                                                                                                                                                                                                         │
│ -A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff                                                                                                                                                                                                                                                     │
│ -A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff                                                                                                                                                                                                                 │
│ COMMIT                                                                                                                                                                                                                                                                                                                            │
│ * nat                                                                                                                                                                                                                                                                                                                             │
│ -N ISTIO_PRERT                                                                                                                                                                                                                                                                                                                    │
│ -N ISTIO_OUTPUT                                                                                                                                                                                                                                                                                                                   │
│ -A OUTPUT -j ISTIO_OUTPUT                                                                                                                                                                                                                                                                                                         │
│ -A PREROUTING -j ISTIO_PRERT                                                                                                                                                                                                                                                                                                      │
│ -A ISTIO_PRERT -s 169.254.7.127 -p tcp -m tcp -j ACCEPT                                                                                                                                                                                                                                                                           │
│ -A ISTIO_OUTPUT -d 169.254.7.127 -p tcp -m tcp -j ACCEPT                                                                                                                                                                                                                                                                          │
│ -A ISTIO_PRERT ! -d 127.0.0.1/32 -p tcp ! --dport 15008 -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15006                                                                                                                                                                                                                 │
│ -A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT                                                                                                                                                                                                                                                                       │
│ -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -o lo -j ACCEPT                                                                                                                                                                                                                                                                                 │
│ -A ISTIO_OUTPUT ! -d 127.0.0.1/32 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001                                                                                                                                                                                                                                │
│ COMMIT    ns=pepr-system name=pepr-uds-core-75b945f5fd-6jdj2                                                                                                                                                                                                                                                                      │
│ 2024-12-12T00:16:03.425283Z    info    Running command (without nss): iptables-nft-restore --noflush -v                                                                                                                                                                                                                           │
│ 2024-12-12T00:16:03.427899Z    info    cni-agent    Running ip6tables-nft-restore with the following input:                                                                                                                                                                                                                       │
│ * mangle                                                                                                                                                                                                                                                                                                                          │
│ -N ISTIO_PRERT                                                                                                                                                                                                                                                                                                                    │
│ -N ISTIO_OUTPUT                                                                                                                                                                                                                                                                                                                   │
│ -A PREROUTING -j ISTIO_PRERT                                                                                                                                                                                                                                                                                                      │
│ -A OUTPUT -j ISTIO_OUTPUT                                                                                                                                                                                                                                                                                                         │
│ -A ISTIO_PRERT -m mark --mark 0x539/0xfff -j CONNMARK --set-xmark 0x111/0xfff                                                                                                                                                                                                                                                     │
│ -A ISTIO_OUTPUT -m connmark --mark 0x111/0xfff -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff                                                                                                                                                                                                                 │
│ COMMIT                                                                                                                                                                                                                                                                                                                            │
│ * nat                                                                                                                                                                                                                                                                                                                             │
│ -N ISTIO_PRERT                                                                                                                                                                                                                                                                                                                    │
│ -N ISTIO_OUTPUT                                                                                                                                                                                                                                                                                                                   │
│ -A OUTPUT -j ISTIO_OUTPUT                                                                                                                                                                                                                                                                                                         │
│ -A PREROUTING -j ISTIO_PRERT                                                                                                                                                                                                                                                                                                      │
│ -A ISTIO_PRERT -s fd16:9254:7127:1337:ffff:ffff:ffff:ffff -p tcp -m tcp -j ACCEPT                                                                                                                                                                                                                                                 │
│ -A ISTIO_OUTPUT -d fd16:9254:7127:1337:ffff:ffff:ffff:ffff -p tcp -m tcp -j ACCEPT                                                                                                                                                                                                                                                │
│ -A ISTIO_PRERT ! -d ::1/128 -p tcp ! --dport 15008 -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15006                                                                                                                                                                                                                      │
│ -A ISTIO_OUTPUT -p tcp -m mark --mark 0x111/0xfff -j ACCEPT                                                                                                                                                                                                                                                                       │
│ -A ISTIO_OUTPUT ! -d ::1/128 -o lo -j ACCEPT                                                                                                                                                                                                                                                                                      │
│ -A ISTIO_OUTPUT ! -d ::1/128 -p tcp -m mark ! --mark 0x539/0xfff -j REDIRECT --to-ports 15001                                                                                                                                                                                                                                     │
│ COMMIT    ns=pepr-system name=pepr-uds-core-75b945f5fd-6jdj2                                                                                                                                                                                                                                                                      │
│ 2024-12-12T00:16:03.427933Z    info    Running command (without nss): ip6tables-nft-restore --noflush -v                                                                                                                                                                                                                          │
│ 2024-12-12T00:16:03.430027Z    info    cni-agent    sending pod add to ztunnel    uid=290bd8af-3088-4a8a-a543-863afb494377 name=pepr-uds-core-75b945f5fd-6jdj2 namespace=pepr-system serviceAccount=pepr-uds-core                                                                                                                 │
│ 2024-12-12T00:16:03.446426Z    error    cni-agent    failed adding pod pepr-uds-core-75b945f5fd-6jdj2 to ipset %!s(*string=0xc000260b38) with ip 10.42.0.215, error was failed to add IP 10.42.0.215 to ipset istio-inpod-probes-v4: invalid comment                                                                              │
│ 2024-12-12T00:16:03.446440Z    error    cni-agent    failed to add pod to ipset: pepr-system/pepr-uds-core-75b945f5fd-6jdj2 failed to add IP 10.42.0.215 to ipset istio-inpod-probes-v4: invalid comment                                                                                                                          │
│ 2024-12-12T00:16:03.446446Z    error    cni-agent    failed to handle add event: failed to add IP 10.42.0.215 to ipset istio-inpod-probes-v4: invalid comment    ns=pepr-system name=pepr-uds-core-75b945f5fd-6jdj2                                                                                                               │
│ 2024-12-12T00:16:03.446566Z    error    cni-plugin    istio-cni cmdAdd failed to signal node Istio CNI agent: unable to push CNI event (status code 500): failed to add IP 10.42.0.215 to ipset istio-inpod-probes-v4: invalid comment    pod=pepr-system/pepr-uds-core-75b945f5fd-6jdj2                                          │
│ 2024-12-12T00:16:03.446596Z    error    cni-plugin    istio-cni cmdAdd error: unable to push CNI event (status code 500): failed to add IP 10.42.0.215 to ipset istio-inpod-probes-v4: invalid comment                                                                                                                            │
│ 2024-12-12T00:16:03.446608Z    error    cni-plugin    istio-cni failed with: unable to push CNI event (status code 500): failed to add IP 10.42.0.215 to ipset istio-inpod-probes-v4: invalid comment

Fix is merged but waiting on new release of istio with the fix in it.

[sgettys]

Waiting for the new Istio release to test out on k3s, in the meantime considering what it would take to have 2 functional layers for Base, one with Ambient and one with Sidecar. This could potentially be least disruptive but could also be difficult to make all the layers compatible with both istio deployments and be more maintenance. Investigating what that looks like.
ADR: #1118

[sgettys]

Figuring out how to do L7 routing with Waypoints. Need the following:

• Label namespace with istio.io/use-waypoint:
• Apply Gateway resource, ex:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: waypoint
  namespace: neuvector
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

• Add "istio.io/ingress-use-waypoint: true" to exposed service [ambient] Ingress gateway to waypoint istio/istio#51214
  • Would prefer to label the namespace: istio.io/ingress-use-waypoint applied namespace wide rather than just per service istio/istio#53868