v2.19 issue with IP assignment still present - Service type: LoadBalancer IPs re-assigned when services are updated

[jennweir]

Setup Details

CIS Version: 2.19.0
Build: 0.289.5
BIGIP Version: Big IP 17.1.1.3
AS3 Version: 3.46.2
Agent Mode: AS3/CCCL
Orchestration: K8S/OSCP
Orchestration Version: 1.28.9 / 4.15.12
Pool Mode: Cluster/Nodeport
Additional Setup details: <Platform/CNI Plugins/ cluster nodes/ etc>

Description

See issue #3615
Bug is still present after upgrading to 2.19

Steps To Reproduce

See below

[ArthurVardevanyan]

Some Additional Testing Notes:

Create Some LBS

for i in {0..40}; do
  yq -i e '.metadata.name="http-echo'"${i}"'"' "f5-bigip-cis/smoke-test/http-echo/base/service.yaml"
  kubectl apply -f f5-bigip-cis/smoke-test/http-echo/base/service.yaml -n smoke-test
done

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE         SELECTOR
http-echo0           LoadBalancer   172.30.80.253    X.X.35.12   8080:32601/TCP,4789:32653/TCP   2m41s       app=http-echo
http-echo1           LoadBalancer   172.30.215.87    X.X.35.13   8080:30518/TCP,4789:30390/TCP   2m41s       app=http-echo
http-echo2           LoadBalancer   172.30.211.173   X.X.35.14   8080:32500/TCP,4789:31956/TCP   2m41s       app=http-echo
http-echo3           LoadBalancer   172.30.34.192    X.X.35.15   8080:32039/TCP,4789:31460/TCP   2m41s       app=http-echo
http-echo4           LoadBalancer   172.30.127.202   X.X.35.16   8080:30285/TCP,4789:32062/TCP   2m41s       app=http-echo
http-echo5           LoadBalancer   172.30.50.139    X.X.35.17   8080:32592/TCP,4789:31988/TCP   2m41s       app=http-echo
http-echo6           LoadBalancer   172.30.187.42    X.X.35.18   8080:30303/TCP,4789:32503/TCP   2m41s       app=http-echo
http-echo7           LoadBalancer   172.30.190.63    X.X.35.19   8080:30524/TCP,4789:31245/TCP   2m41s       app=http-echo
http-echo8           LoadBalancer   172.30.192.113   X.X.35.2    8080:31605/TCP,4789:32397/TCP   2m41s       app=http-echo
http-echo9           LoadBalancer   172.30.125.70    X.X.35.20   8080:30570/TCP,4789:32647/TCP   2m41s       app=http-echo
http-echo10          LoadBalancer   172.30.66.19     X.X.35.21   8080:32434/TCP,4789:31596/TCP   2m41s       app=http-echo
http-echo11          LoadBalancer   172.30.67.153    X.X.35.22   8080:31124/TCP,4789:32308/TCP   2m41s       app=http-echo
http-echo12          LoadBalancer   172.30.161.171   X.X.35.23   8080:30930/TCP,4789:30557/TCP   2m41s       app=http-echo
http-echo13          LoadBalancer   172.30.237.187   X.X.35.24   8080:30838/TCP,4789:31521/TCP   2m40s       app=http-echo
http-echo14          LoadBalancer   172.30.48.36     X.X.35.25   8080:32764/TCP,4789:31944/TCP   2m39s       app=http-echo
http-echo15          LoadBalancer   172.30.28.241    X.X.35.26   8080:32458/TCP,4789:30246/TCP   2m38s       app=http-echo
http-echo16          LoadBalancer   172.30.23.97     X.X.35.27   8080:32603/TCP,4789:30301/TCP   2m37s       app=http-echo
http-echo17          LoadBalancer   172.30.44.175    X.X.35.28   8080:31818/TCP,4789:32373/TCP   2m36s       app=http-echo
http-echo18          LoadBalancer   172.30.145.184   X.X.35.29   8080:32063/TCP,4789:31252/TCP   2m36s       app=http-echo
http-echo19          LoadBalancer   172.30.202.176   X.X.35.3    8080:32325/TCP,4789:30691/TCP   2m35s       app=http-echo
http-echo20          LoadBalancer   172.30.169.120   X.X.35.30   8080:31857/TCP,4789:32670/TCP   2m34s       app=http-echo
http-echo21          LoadBalancer   172.30.215.103   X.X.35.31   8080:32631/TCP,4789:31678/TCP   2m34s       app=http-echo
http-echo22          LoadBalancer   172.30.254.235   X.X.35.32   8080:32038/TCP,4789:30392/TCP   2m33s       app=http-echo
http-echo23          LoadBalancer   172.30.41.33     X.X.35.33   8080:31766/TCP,4789:30332/TCP   2m32s       app=http-echo
http-echo24          LoadBalancer   172.30.44.105    X.X.35.34   8080:31235/TCP,4789:30610/TCP   2m31s       app=http-echo
http-echo25          LoadBalancer   172.30.1.98      X.X.35.35   8080:30305/TCP,4789:31482/TCP   2m30s       app=http-echo
http-echo26          LoadBalancer   172.30.150.14    X.X.35.36   8080:31842/TCP,4789:32022/TCP   2m29s       app=http-echo
http-echo27          LoadBalancer   172.30.210.10    X.X.35.37   8080:31600/TCP,4789:32568/TCP   2m29s       app=http-echo
http-echo28          LoadBalancer   172.30.33.48     X.X.35.38   8080:31961/TCP,4789:30808/TCP   2m28s       app=http-echo
http-echo29          LoadBalancer   172.30.160.169   X.X.35.39   8080:31294/TCP,4789:31169/TCP   2m27s       app=http-echo
http-echo30          LoadBalancer   172.30.168.115   X.X.35.4    8080:30127/TCP,4789:32536/TCP   2m27s       app=http-echo
http-echo31          LoadBalancer   172.30.106.173   X.X.35.40   8080:30084/TCP,4789:30526/TCP   2m26s       app=http-echo
http-echo32          LoadBalancer   172.30.255.19    X.X.35.41   8080:30245/TCP,4789:32720/TCP   2m25s       app=http-echo
http-echo33          LoadBalancer   172.30.211.161   X.X.35.42   8080:30702/TCP,4789:30291/TCP   2m24s       app=http-echo
http-echo34          LoadBalancer   172.30.229.255   X.X.35.43   8080:31157/TCP,4789:32287/TCP   2m24s       app=http-echo
http-echo35          LoadBalancer   172.30.147.86    X.X.35.44   8080:30952/TCP,4789:31283/TCP   2m23s       app=http-echo
http-echo36          LoadBalancer   172.30.17.27     X.X.35.45   8080:30094/TCP,4789:31198/TCP   2m21s       app=http-echo
http-echo37          LoadBalancer   172.30.100.224   X.X.35.46   8080:31015/TCP,4789:32236/TCP   2m20s       app=http-echo
http-echo38          LoadBalancer   172.30.181.221   X.X.35.47   8080:30854/TCP,4789:31395/TCP   2m20s       app=http-echo
http-echo39          LoadBalancer   172.30.245.163   X.X.35.48   8080:31196/TCP,4789:32645/TCP   2m19s       app=http-echo
http-echo40          LoadBalancer   172.30.156.189   X.X.35.49   8080:30427/TCP,4789:30451/TCP   2m18s       app=http-echo
test-http-echo-svc   LoadBalancer   172.30.101.71    X.X.35.11   8080:31602/TCP,4789:32352/TCP   24m         app=http-echo

Delete first 10 LBs

for i in {0..10}; do
  yq -i e '.metadata.name="http-echo'"${i}"'"' "f5-bigip-cis/smoke-test/http-echo/base/service.yaml"
  kubectl delete -f f5-bigip-cis/smoke-test/http-echo/base/service.yaml -n smoke-test
done

Patch Remaning Service Objects

for svc in $(kubectl get services -n smoke-test -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl patch service $svc -n smoke-test -p '{"metadata": {"labels": {"new-label": "value"}, "annotations": {"new-annotation": "value"}}}'
done

NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE     SELECTOR
http-echo11          LoadBalancer   172.30.67.153    X.X.35.12   8080:31124/TCP,4789:32308/TCP   7m      app=http-echo
http-echo12          LoadBalancer   172.30.161.171   X.X.35.13   8080:30930/TCP,4789:30557/TCP   7m      app=http-echo
http-echo13          LoadBalancer   172.30.237.187   X.X.35.14   8080:30838/TCP,4789:31521/TCP   6m59s   app=http-echo
http-echo14          LoadBalancer   172.30.48.36     X.X.35.15   8080:32764/TCP,4789:31944/TCP   6m58s   app=http-echo
http-echo15          LoadBalancer   172.30.28.241    X.X.35.16   8080:32458/TCP,4789:30246/TCP   6m57s   app=http-echo
http-echo16          LoadBalancer   172.30.23.97     X.X.35.17   8080:32603/TCP,4789:30301/TCP   6m56s   app=http-echo
http-echo17          LoadBalancer   172.30.44.175    X.X.35.18   8080:31818/TCP,4789:32373/TCP   6m55s   app=http-echo
http-echo18          LoadBalancer   172.30.145.184   X.X.35.19   8080:32063/TCP,4789:31252/TCP   6m55s   app=http-echo
http-echo19          LoadBalancer   172.30.202.176   X.X.35.2    8080:32325/TCP,4789:30691/TCP   6m54s   app=http-echo
http-echo20          LoadBalancer   172.30.169.120   X.X.35.20   8080:31857/TCP,4789:32670/TCP   6m53s   app=http-echo
http-echo21          LoadBalancer   172.30.215.103   X.X.35.21   8080:32631/TCP,4789:31678/TCP   6m53s   app=http-echo
http-echo22          LoadBalancer   172.30.254.235   X.X.35.22   8080:32038/TCP,4789:30392/TCP   6m52s   app=http-echo
http-echo23          LoadBalancer   172.30.41.33     X.X.35.23   8080:31766/TCP,4789:30332/TCP   6m51s   app=http-echo
http-echo24          LoadBalancer   172.30.44.105    X.X.35.11   8080:31235/TCP,4789:30610/TCP   6m50s   app=http-echo
http-echo25          LoadBalancer   172.30.1.98      X.X.35.24   8080:30305/TCP,4789:31482/TCP   6m49s   app=http-echo
http-echo26          LoadBalancer   172.30.150.14    X.X.35.25   8080:31842/TCP,4789:32022/TCP   6m48s   app=http-echo
http-echo27          LoadBalancer   172.30.210.10    X.X.35.26   8080:31600/TCP,4789:32568/TCP   6m48s   app=http-echo
http-echo28          LoadBalancer   172.30.33.48     X.X.35.27   8080:31961/TCP,4789:30808/TCP   6m47s   app=http-echo
http-echo29          LoadBalancer   172.30.160.169   X.X.35.28   8080:31294/TCP,4789:31169/TCP   6m46s   app=http-echo
http-echo30          LoadBalancer   172.30.168.115   X.X.35.29   8080:30127/TCP,4789:32536/TCP   6m46s   app=http-echo
http-echo31          LoadBalancer   172.30.106.173   X.X.35.3    8080:30084/TCP,4789:30526/TCP   6m45s   app=http-echo
http-echo32          LoadBalancer   172.30.255.19    X.X.35.30   8080:30245/TCP,4789:32720/TCP   6m44s   app=http-echo
http-echo33          LoadBalancer   172.30.211.161   X.X.35.31   8080:30702/TCP,4789:30291/TCP   6m43s   app=http-echo
http-echo34          LoadBalancer   172.30.229.255   X.X.35.32   8080:31157/TCP,4789:32287/TCP   6m43s   app=http-echo
http-echo35          LoadBalancer   172.30.147.86    X.X.35.33   8080:30952/TCP,4789:31283/TCP   6m42s   app=http-echo
http-echo36          LoadBalancer   172.30.17.27     X.X.35.34   8080:30094/TCP,4789:31198/TCP   6m40s   app=http-echo
http-echo37          LoadBalancer   172.30.100.224   X.X.35.35   8080:31015/TCP,4789:32236/TCP   6m39s   app=http-echo
http-echo38          LoadBalancer   172.30.181.221   X.X.35.36   8080:30854/TCP,4789:31395/TCP   6m39s   app=http-echo
http-echo39          LoadBalancer   172.30.245.163   X.X.35.37   8080:31196/TCP,4789:32645/TCP   6m38s   app=http-echo
http-echo40          LoadBalancer   172.30.156.189   X.X.35.38   8080:30427/TCP,4789:30451/TCP   6m37s   app=http-echo
test-http-echo-svc   LoadBalancer   172.30.101.71    X.X.35.39   8080:31602/TCP,4789:32352/TCP   29m     app=http-echo

**

[trinaths]

@ArthurVardevanyan @jennweir Please share CIS configuration and debug logs, example manifests to [email protected]

[trinaths]

Created [CONTCNTR-5193] for internal tracking.

[ArthurVardevanyan]

@ArthurVardevanyan @jennweir Please share CIS configuration and debug logs, example manifests to [email protected]

Sent Over, Thank You

[trinaths]

Created [CONTCNTR-5193] for internal tracking.