axgbe: traffic stall for 1G modules on link-up after boottime interface configuration

[swhite2]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/src/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/src/issues?q=is%3Aissue

Describe the bug

1G SFP modules that actively auto-negotiate on axgbe are rather slow to come up in link state, which can cause a race condition as described in opnsense/core#7113 (comment)

However, one would expect that after boot, removing the SFP and inserting it again would eventually lead to dhclient recovering and fetching an IP address for the interface after the link up. Instead, the driver remains in an "off" state, where only out traffic can be seen on tcpdump, and the device does not receive any traffic.

This issue has only been observed with (slow) 1G modules, which suggests that there is still an issue in the driver wrt the link up flow. Interfaces that have a static IP assigned to them are not affected. The issue is also unlikely to be reproducible on all 1G SFP modules.

To Reproduce

Steps to reproduce the behavior:

1. With a 1G module inserted, assign ax0 as WAN interface.
2. Reboot the machine.
3. After boot, there is no IP address on the WAN interface.
4. Remove & insert SFP, link up event coalescing from devd will fire, and dhclient will start, but will not settle as a tcpdump will reveal no incoming traffic.
5. sysctl dev.ax.0 will show no activity on the receive queues.
6. ifconfig ax0 down && ifconfig ax0 up
7. WAN interface will get an IP address.

Expected behavior

For axgbe to receive traffic after a link up after boottime interface configuration.

Describe alternatives you considered

A manual ifconfig ax<N> down && ifconfig ax<N> up fixes this. As a workaround, one can create a script in /usr/local/etc/rc.syshook.d/start named 90-axgbe which contains the above down & up.