debian instance not accessible when multiple network interfaces present

[rcgoodfellow]

I created an instance with a Debian 12 cloud-init image with 3 network interfaces. When it came up, I was not able to access it via its external IP.

This is what it looks like from the CLI.

$ oxide instance view --instance cr0 --project router-testing
success
Instance {
    description: "Customer router zero",
    hostname: "cr0",
    id: dc0a52a1-db1c-4717-b3ac-28c7b106cad2,
    memory: ByteCount(
        8589934592,
    ),
    name: Name(
        "cr0",
    ),
    ncpus: InstanceCpuCount(
        4,
    ),
    project_id: 4d4a8023-a8c0-4c13-96b5-71cd04bd4402,
    run_state: Running,
    time_created: 2024-01-18T02:08:16.004264Z,
    time_modified: 2024-01-18T02:08:16.004264Z,
    time_run_state_updated: 2024-01-18T02:08:50.834602Z,
}

$ oxide instance nic list --instance cr0 --project router-testing
InstanceNetworkInterface {
    description: "",
    id: 0ce1343a-a174-4d68-8853-982f0b047024,
    instance_id: dc0a52a1-db1c-4717-b3ac-28c7b106cad2,
    ip: 172.30.0.9,
    mac: MacAddr(
        "A8:40:25:FD:F3:63",
    ),
    name: Name(
        "mgmt",
    ),
    primary: true,
    subnet_id: 9a1864b4-440b-428d-9e9e-0b949d7d1fe7,
    time_created: 2024-01-18T02:08:16.214846Z,
    time_modified: 2024-01-18T02:08:16.214846Z,
    vpc_id: f1db67cd-b1fd-49ef-8401-4c6691151e21,
}
InstanceNetworkInterface {
    description: "",
    id: f5b2572c-a7ce-4d45-9dff-d36bc528e841,
    instance_id: dc0a52a1-db1c-4717-b3ac-28c7b106cad2,
    ip: 10.0.0.10,
    mac: MacAddr(
        "A8:40:25:FC:75:BE",
    ),
    name: Name(
        "sw0",
    ),
    primary: false,
    subnet_id: b0703d91-4097-49b1-a7b6-197be5b108ca,
    time_created: 2024-01-18T02:08:16.433674Z,
    time_modified: 2024-01-18T02:08:16.433674Z,
    vpc_id: f1db67cd-b1fd-49ef-8401-4c6691151e21,
}
InstanceNetworkInterface {
    description: "",
    id: 1a51f3d0-8911-4ab0-9dbd-b0b8c6e0ca65,
    instance_id: dc0a52a1-db1c-4717-b3ac-28c7b106cad2,
    ip: 10.0.1.10,
    mac: MacAddr(
        "A8:40:25:F8:74:4B",
    ),
    name: Name(
        "sw1",
    ),
    primary: false,
    subnet_id: d4bc0098-ab25-4315-911e-46984d40558c,
    time_created: 2024-01-18T02:08:16.645434Z,
    time_modified: 2024-01-18T02:08:16.645434Z,
    vpc_id: f1db67cd-b1fd-49ef-8401-4c6691151e21,
}

From the instance console output, it looks like the default route is being assigned the wrong gateway.

[    7.072788] cloud-init[424]: ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
[    7.076163] cloud-init[424]: ci-info: +-------+-------------+----------+-----------------+-----------+-------+
[    7.079531] cloud-init[424]: ci-info: | Route | Destination | Gateway  |     Genmask     | Interface | Flags |
[    7.082890] cloud-init[424]: ci-info: +-------+-------------+----------+-----------------+-----------+-------+
[    7.086241] cloud-init[424]: ci-info: |   0   |   0.0.0.0   | 10.0.1.1 |     0.0.0.0     |   enp0s8  |   UG  |
[    7.089635] cloud-init[424]: ci-info: |   1   |   1.1.1.1   | 10.0.1.1 | 255.255.255.255 |   enp0s8  |  UGH  |
[    7.092972] cloud-init[424]: ci-info: |   2   |   8.8.8.8   | 10.0.1.1 | 255.255.255.255 |   enp0s8  |  UGH  |
[    7.096323] cloud-init[424]: ci-info: |   3   |   10.0.1.1  | 0.0.0.0  | 255.255.255.255 |   enp0s8  |   UH  |
[    7.099669] cloud-init[424]: ci-info: +-------+-------------+----------+-----------------+-----------+-------+

It also looks like some address assignments are missing.

[    7.027556] cloud-init[424]: ci-info: +++++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++++
[    7.031646] cloud-init[424]: ci-info: +---------+-------+------------------------------+-----------------+--------+-------------------+
[    7.035839] cloud-init[424]: ci-info: |  Device |   Up  |           Address            |       Mask      | Scope  |     Hw-Address    |
[    7.039945] cloud-init[424]: ci-info: +---------+-------+------------------------------+-----------------+--------+-------------------+
[    7.044096] cloud-init[424]: ci-info: | enp0s10 | False |              .               |        .        |   .    | a8:40:25:fc:75:be |
[    7.048215] cloud-init[424]: ci-info: |  enp0s8 |  True |          10.0.1.10           | 255.255.255.255 | global | a8:40:25:f8:74:4b |
[    7.052324] cloud-init[424]: ci-info: |  enp0s8 |  True | fe80::aa40:25ff:fef8:744b/64 |        .        |  link  | a8:40:25:f8:74:4b |
[    7.056441] cloud-init[424]: ci-info: |  enp0s9 | False |              .               |        .        |   .    | a8:40:25:fd:f3:63 |
[    7.060527] cloud-init[424]: ci-info: |    lo   |  True |          127.0.0.1           |    255.0.0.0    |  host  |         .         |
[    7.064617] cloud-init[424]: ci-info: |    lo   |  True |           ::1/128            |        .        |  host  |         .         |
[    7.068703] cloud-init[424]: ci-info: +---------+-------+------------------------------+-----------------+--------+-------------------+

[rcgoodfellow]

Stopping the instance, removing the non-primary interfaces and starting again lead a different state, but still no connectivity. Console output shows this from cloud init

[  124.030410] cloud-init[403]: Cloud-init v. 22.4.2 running 'init' at Thu, 18 Jan 2024 02:31:39 +0000. Up 124.01 seconds.
[  124.038961] cloud-init[403]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
[  124.042813] cloud-init[403]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[  124.046204] cloud-init[403]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
[  124.049642] cloud-init[403]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
[  124.053085] cloud-init[403]: ci-info: | enp0s9 | False |     .     |     .     |   .   | a8:40:25:fd:f3:63 |
[  124.056512] cloud-init[403]: ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |  host |         .         |
[  124.059935] cloud-init[403]: ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |
[  124.063374] cloud-init[403]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+

I'm not able to log into this instance without network connectivity or I would do so and dig in more.

[askfongjojo]

I've been trying out how different guest OS and versions work with multiple nics and they all seem to behave differently when it comes to the default route settings. The observations so far are summarized as follows:

• The default route is not tied to the primary nic designation. They may coincide by chance (e.g. if you specify that primary nic first at instance creation time and add other nics later on).
• In the newer debian and ubuntu versions, only the nic in slot 0 is configured in the guest's nic and route settings, as seen in the description of this ticket. (Note: If the propolis has a new nic added subsequently to swap out the old nic, because it's given the next slot number, i.e. slot 1 as stored in CRDB, the instance will come up with no nic/route.)
• In centos and debian 11, they look like the following when there are multiple nics:

[    7.246047] cloud-init[834]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++
[    7.249179] cloud-init[834]: ci-info: +--------+------+-----------------------------+-----------------+--------+-------------------+
[    7.252255] cloud-init[834]: ci-info: | Device |  Up  |           Address           |       Mask      | Scope  |     Hw-Address    |
[    7.255326] cloud-init[834]: ci-info: +--------+------+-----------------------------+-----------------+--------+-------------------+
[    7.258414] cloud-init[834]: ci-info: |  eth0  | True |         192.168.32.8        | 255.255.255.255 | global | a8:40:25:fe:07:7b |
[    7.261489] cloud-init[834]: ci-info: |  eth0  | True | fe80::aa40:25ff:fefe:77b/64 |        .        |  link  | a8:40:25:fe:07:7b |
[    7.264552] cloud-init[834]: ci-info: |  eth1  | True |          172.30.0.7         | 255.255.255.255 | global | a8:40:25:f9:0d:f3 |
[    7.267621] cloud-init[834]: ci-info: |  eth1  | True | fe80::95a8:83c3:d080:7bd/64 |        .        |  link  | a8:40:25:f9:0d:f3 |
[    7.270705] cloud-init[834]: ci-info: |   lo   | True |          127.0.0.1          |    255.0.0.0    |  host  |         .         |
[    7.273773] cloud-init[834]: ci-info: |   lo   | True |           ::1/128           |        .        |  host  |         .         |
[    7.276834] cloud-init[834]: ci-info: +--------+------+-----------------------------+-----------------+--------+-------------------+
[    7.279903] cloud-init[834]: ci-info: +++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
[    7.282608] cloud-init[834]: ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
[    7.285319] cloud-init[834]: ci-info: | Route | Destination  |   Gateway    |     Genmask     | Interface | Flags |
[    7.288042] cloud-init[834]: ci-info: +-------+--------------+--------------+-----------------+-----------+-------+
[    7.290747] cloud-init[834]: ci-info: |   0   |   0.0.0.0    | 192.168.32.1 |     0.0.0.0     |    eth0   |   UG  |
[    7.293471] cloud-init[834]: ci-info: |   1   |   0.0.0.0    |  172.30.0.1  |     0.0.0.0     |    eth1   |   UG  |
[    7.296181] cloud-init[834]: ci-info: |   2   |  172.30.0.1  |   0.0.0.0    | 255.255.255.255 |    eth1   |   UH  |
[    7.298885] cloud-init[834]: ci-info: |   3   | 192.168.32.1 |   0.0.0.0    | 255.255.255.255 |    eth0   |   UH  |
[    7.301594] cloud-init[834]: ci-info: +-------+--------------+--------------+-----------------+-----------+-------+

• The behavior is largely driven by cloud-init and it doesn't appear that our platform attempts to pass network interface specs to the 50-cloud-init based on the instance nic data.

The question here is - should we leave the net assignment to user (i.e. for them to configure cloud-init to achieve what they want)? The answer seems to be "no" because we explicitly provide a way for user to designate the primary nic.

@pfmooney, thoughts?

[askfongjojo]

I should also mention that I've inspected the opte entries on the sled in some of the cases when I couldn't SSH in because the default route was on the non-primary nic. As I had set the password of the default user, I accessed the VM via serial console and removed the incorrect default route from there, e.g.,

cloud-user@nic1 ~]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.32.1    0.0.0.0         UG    100    0        0 eth0
0.0.0.0         172.30.0.1      0.0.0.0         UG    101    0        0 eth1
172.30.0.1      0.0.0.0         255.255.255.255 UH    101    0        0 eth1
192.168.32.1    0.0.0.0         255.255.255.255 UH    100    0        0 eth0
[cloud-user@nic1 ~]$ sudo ip route del default
[cloud-user@nic1 ~]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.30.0.1      0.0.0.0         UG    101    0        0 eth1
172.30.0.1      0.0.0.0         255.255.255.255 UH    101    0        0 eth1
192.168.32.1    0.0.0.0         255.255.255.255 UH    100    0        0 eth0

Once the bad route was taken out, I was able to SSH into the VM on the external IP (which is NAT-ed to the 172.30.0.x primary nic). All that is to say the opte side of things is working as expected so what remains unclear is the plumbing into the guest OS. I tried digging into the propolis zone settings and all I could tell was that the nics were passed but there was nothing that marked an interface as primary:

BRM44220011 # zonecfg -z oxz_propolis-server_e5894206-fe5f-4d3c-8893-9a41d04d4716 info
zonename: oxz_propolis-server_e5894206-fe5f-4d3c-8893-9a41d04d4716
zonepath: /pool/ext/cf940e15-dbc5-481b-866a-4de4b018898e/crypt/zone/oxz_propolis-server_e5894206-fe5f-4d3c-8893-9a41d04d4716
brand: omicron1
autoboot: false
bootargs: 
pool: 
limitpriv: 
scheduling-class: 
ip-type: exclusive
hostid: 
fs-allowed: 
net:
	address not specified
	allowed-address not specified
	physical: opte7
	defrouter not specified
net:
	address not specified
	allowed-address not specified
	physical: opte8
	defrouter not specified
net:
	address not specified
	allowed-address not specified
	physical: oxControlInstance6
	defrouter not specified
device:
	match: /dev/vmm/*
device:
	match: /dev/vmmctl
device:
	match: /dev/viona

[wfchandler]

I ran into this when deleting the network interface on an existing VM and creating a new one in a different VPC.

The console showed that an IP address had been assigned to the host, but the link was down.

ubuntu@vpc-change-test:~$ ip a
1: lo: \<LOOPBACK,UP,LOWER\_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid\_lft forever preferred\_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid\_lft forever preferred\_lft forever
2: enp0s8: \<BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether a8:40:25:f0:00:01 brd ff:ff:ff:ff:ff:ff

After enabling the link and rebinding DHCP, the connection on the new VPC was established:

ubuntu@vpc-change-test:~$ sudo ip link set up dev enp0s8
ubuntu@vpc-change-test:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether a8:40:25:f0:00:01 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::aa40:25ff:fef0:1/64 scope link 
       valid_lft forever preferred_lft forever

ubuntu@vpc-change-test:~$ sudo dhcpcd -n enp0s8
dhcpcd-10.0.6 starting
DUID 00:01:00:01:2e:a2:91:91:a8:40:25:f0:00:01
enp0s8: IAID 25:f0:00:01
enp0s8: soliciting an IPv6 router
enp0s8: soliciting a DHCP lease
enp0s8: offered 172.30.0.6 from 172.30.0.1
enp0s8: probing address 172.30.0.6/32
enp0s8: leased 172.30.0.6 for 86400 seconds
enp0s8: adding host route to 172.30.0.1
enp0s8: adding default route via 172.30.0.1
Dropped protocol specifier '.dhcp' from 'enp0s8.dhcp'. Using 'enp0s8' (ifindex=2).
ubuntu@vpc-change-test:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether a8:40:25:f0:00:01 brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.6/32 brd 172.30.0.6 scope global dynamic enp0s8
       valid_lft 86392sec preferred_lft 75592sec
    inet6 fe80::aa40:25ff:fef0:1/64 scope link 
       valid_lft forever preferred_lft forever

[FelixMcFelix]

Some of the OPTE state does appear to be a little messed up here. I set up an instance with two NICs on dogfood, then swapped them and set the newly created one as primary. The original NIC is in the subnet 10.0.1.0/24, and the new primary is in 192.168.0.0/24.

BRM42220006 # opteadm list-ports
LINK    MAC ADDRESS        IPv4 ADDRESS  EPHEMERAL IPv4  FLOATING IPv4  IPv6 ADDRESS  EXTERNAL IPv6  FLOATING IPv6  STATE
...
opte19  A8:40:25:FB:49:BC  10.0.1.6      172.20.26.114   None           None          None           None           running
opte20  A8:40:25:F3:9E:39  192.168.0.6   172.20.26.114   None           None          None           None           running

BRM42220006 # opteadm dump-layer -p opte19 nat
Port opte19 - Layer nat
======================================================================
...

Inbound Rules
----------------------------------------------------------------------
ID   PRI  HITS  PREDICATES                  ACTION
0    10   0     inner.ip.dst=172.20.26.114  "Stateful: 10.0.1.6 <=> (external)"
DEF  --   0     --                          "allow"

Outbound Rules
----------------------------------------------------------------------
ID   PRI  HITS  PREDICATES                   ACTION
0    10   255   inner.ether.ether_type=IPv4  "Stateful: 10.0.1.6 <=> 172.20.26.114"
                meta: router-target=ig

1    100  0     inner.ether.ether_type=IPv4  "Stateful: 172.20.26.27:0-16383"
                meta: router-target=ig

DEF  --   0     --                           "allow"

BRM42220006 # opteadm dump-layer -p opte20 nat
Port opte20 - Layer nat
======================================================================
...

Inbound Rules
----------------------------------------------------------------------
ID   PRI  HITS  PREDICATES                  ACTION
0    10   259   inner.ip.dst=172.20.26.114  "Stateful: 192.168.0.6 <=> (external)"
DEF  --   0     --                          "allow"

Outbound Rules
----------------------------------------------------------------------
ID   PRI  HITS  PREDICATES                   ACTION
0    10   0     inner.ether.ether_type=IPv4  "Stateful: 192.168.0.6 <=> 172.20.26.114"
                meta: router-target=ig

1    100  0     inner.ether.ether_type=IPv4  "Stateful: 172.20.26.27:0-16383"
                meta: router-target=ig

DEF  --   0     --                           "allow"

Note that opte19 is the original NIC, while opte20 is the new primary. So both NICs are being assigned the same Ephemeral and SNAT IP. What if we add a floating IP dynamically?

BRM42220006 # opteadm list-ports
LINK    MAC ADDRESS        IPv4 ADDRESS  EPHEMERAL IPv4  FLOATING IPv4  IPv6 ADDRESS  EXTERNAL IPv6  FLOATING IPv6  STATE
opte0   A8:40:25:FF:B0:9C  172.30.1.5    None            172.20.26.1    None          None           None           running
opte3   A8:40:25:F4:45:B7  172.30.0.6    172.20.26.170   None           None          None           None           running
opte5   A8:40:25:F2:9C:6F  172.30.0.9    172.20.26.25    None           None          None           None           running
opte7   A8:40:25:F0:54:A7  172.30.0.20   None            None           None          None           None           running
opte8   A8:40:25:F2:B6:C8  172.30.0.12   172.20.26.35    None           None          None           None           running
opte16  A8:40:25:F0:B9:67  172.30.0.14   172.20.26.102   None           None          None           None           running
opte19  A8:40:25:FB:49:BC  10.0.1.6      172.20.26.114   172.20.26.184  None          None           None           running
opte20  A8:40:25:F3:9E:39  192.168.0.6   172.20.26.114   None           None          None           None           running

It's being assigned to opte19, which is clearly not the primary NIC. Finally, the NAT bindings look a-okay. We want them to map 172.20.26.27:0--16383, 172.20.26.114, 172.20.26.184 to the MAC address A8:40:25:F3:9E:39 on the same sled:

root@oxz_switch0:~# swadm nat list
External IP    Port low  Port high  Internal IP            Inner MAC          VNI
...
172.20.26.27   0         16383      fd00:1122:3344:106::1  a8:40:25:f3:9e:39  1221231
...
172.20.26.114  0         65535      fd00:1122:3344:106::1  a8:40:25:f3:9e:39  1221231
...
172.20.26.184  0         65535      fd00:1122:3344:106::1  a8:40:25:f3:9e:39  1221231

[FelixMcFelix]

The OPTE-related symptoms should be fixed by #6890 -- from poking at it further the issue would only crop up on changes to EIP configuration (i.e., new FIP/Ephemeral added dynamically, or associated IGW tags for existing addresses).