Fix: use correct primary NIC for External IP add/delete/refresh

[FelixMcFelix]

InstanceRunner has been using the wrong primary NIC for Ephemeral/Floating IP addition/removal for some time, assuming that the Slot 0 NIC is always the primary. Now that External IP state is regularly re-asserted to keep Internet Gateway tags up-to-date, this led to OPTE ports reporting the use of external IPs on both the primary and slot0 nic, when multiple NICs are present and external IP state is changed on the instance.

This PR makes the correct NIC selection for all three operations.

[FelixMcFelix]

Testing Notes

An instance was set up with two NICs (with addrs 172.30.0.5 and 192.168.0.5) -- the primary NIC was placed in slot 1. Examining OPTE state once the instance is started:

kyle@farme:~/gits/omicron$ pfexec opteadm list-ports
LINK   MAC ADDRESS        IPv4 ADDRESS  EPHEMERAL IPv4  FLOATING IPv4  IPv6 ADDRESS  EXTERNAL IPv6  FLOATING IPv6  STATE
opte0  A8:40:25:FF:F0:F3  172.30.3.5    None            None           None          None           None           running
opte1  A8:40:25:FF:EA:B6  172.30.2.5    None            10.1.222.3     None          None           None           running
opte2  A8:40:25:FF:B6:CA  172.30.2.6    None            10.1.222.4     None          None           None           running
opte3  A8:40:25:FF:BF:41  172.30.2.7    None            10.1.222.5     None          None           None           running
opte4  A8:40:25:FF:DD:20  172.30.1.6    None            10.1.222.2     None          None           None           running
opte5  A8:40:25:FF:C0:53  172.30.1.5    None            10.1.222.1     None          None           None           running
opte7  A8:40:25:F0:00:00  172.30.0.5    None            None           None          None           None           running
opte8  A8:40:25:F0:00:01  192.168.0.5   10.1.222.12     None           None          None           None           running

Adding a new floating IP to this instance places it on opte8 as expected (and leaves opte7 untouched):

kyle@farme:~/gits/omicron$ pfexec opteadm list-ports
LINK   MAC ADDRESS        IPv4 ADDRESS  EPHEMERAL IPv4  FLOATING IPv4  IPv6 ADDRESS  EXTERNAL IPv6  FLOATING IPv6  STATE
opte0  A8:40:25:FF:F0:F3  172.30.3.5    None            None           None          None           None           running
opte1  A8:40:25:FF:EA:B6  172.30.2.5    None            10.1.222.3     None          None           None           running
opte2  A8:40:25:FF:B6:CA  172.30.2.6    None            10.1.222.4     None          None           None           running
opte3  A8:40:25:FF:BF:41  172.30.2.7    None            10.1.222.5     None          None           None           running
opte4  A8:40:25:FF:DD:20  172.30.1.6    None            10.1.222.2     None          None           None           running
opte5  A8:40:25:FF:C0:53  172.30.1.5    None            10.1.222.1     None          None           None           running
opte7  A8:40:25:F0:00:00  172.30.0.5    None            None           None          None           None           running
opte8  A8:40:25:F0:00:01  192.168.0.5   10.1.222.12     10.1.222.13    None          None           None           running

As does removing the ephemeral IP:

kyle@farme:~/gits/omicron$ pfexec opteadm list-ports
LINK   MAC ADDRESS        IPv4 ADDRESS  EPHEMERAL IPv4  FLOATING IPv4  IPv6 ADDRESS  EXTERNAL IPv6  FLOATING IPv6  STATE
opte0  A8:40:25:FF:F0:F3  172.30.3.5    None            None           None          None           None           running
opte1  A8:40:25:FF:EA:B6  172.30.2.5    None            10.1.222.3     None          None           None           running
opte2  A8:40:25:FF:B6:CA  172.30.2.6    None            10.1.222.4     None          None           None           running
opte3  A8:40:25:FF:BF:41  172.30.2.7    None            10.1.222.5     None          None           None           running
opte4  A8:40:25:FF:DD:20  172.30.1.6    None            10.1.222.2     None          None           None           running
opte5  A8:40:25:FF:C0:53  172.30.1.5    None            10.1.222.1     None          None           None           running
opte7  A8:40:25:F0:00:00  172.30.0.5    None            None           None          None           None           running
opte8  A8:40:25:F0:00:01  192.168.0.5   None            10.1.222.13    None          None           None           running

[internet-diglett]

Looks good to me!