ipam: Fix init flow in case there are sticky ips in the system

[oshoval]

📑 Description

In case the ovnkube-control-plane pod is restarted,
(for example, node restart, upgrade, or any pod restart reason),
the ipam claim Sync function creates an array of all the ipams
belonging to a specific network, and used named allocator AllocateIPs
method to allocate them, so the allocator will reflect the current
cluster state.

The problem is AllocateIPs allows to allocate only one IP per given subnet,
so once it is used for all the IPs in the network at once, it will fail,
causing the control plane to have a crash loop.

Fix it by calling AllocateIPs per each claim on its own.
The claims were already created correctly, so each claim on its
own is safe to call AllocateIPs.

Add unit test to show the relevant assertion.

Fixes #

Additional Information for reviewers

Seen by deleting the pod while there were 2 ipam claims with IPs, from the same NAD
in the system.

✅ Checks

• My code requires changes to the documentation
• if so, I have updated the documentation as required
• My code requires tests
• if so, I have added and/or updated the tests as required
• All the tests have passed in the CI

How to verify it

[oshoval]

Added unit test that shows the current AllocateIPs behavior when trying to allocate several IPs from the same subnet
This behavior is the problematic one for the current Sync logic

[maiqueb]

Looks good.

[maiqueb]

Maybe we could go a step further and add more unit test that asserts that you can indeed invoke sync for multiple claims:

• ipam claim reconciler level:

  ovn-kubernetes/go-controller/pkg/persistentips/allocator_test.go

  Line 110 in cc3c784

  DescribeTable("syncing the IP allocator from the IPAMClaims is successful when provided with", func(ipamClaims ...interface{}) {

• ovn-kubernetes/go-controller/pkg/clustermanager/secondary_network_unit_test.go

  Line 469 in e492123

  ipamClaimWithIPAddr(claimName, namespace, networkName, subnetIP),

  (I think this would reproduce if you simply just pre-provision yet another claim in the claim fake client).

@oshoval

[oshoval]

Addressed comments
Thanks

[qinqon]

I don't see a Sync test, just allocators.

[oshoval]

I don't see a Sync test, just allocators.

There are two unit tests that have it, those
#4823 (comment)

[oshoval]

addressed comments
thanks

[qinqon]

/lgtm

[oshoval]

rebased