Fix race between sled-agent and zone-setup service

[bnaecker]

• Fixes Possible race between sled-agent and self-assembling switch zone #6149
• Most zones run the zone-network-setup once, at startup, with their underlay addresses already provided by the sled-agent. That's not true for the switch zone, which starts with only a localhost address, and then is provided an underlay address by the sled-agent only after the bootstrapping process has proceededed further. However, the zone-setup-service previously deleted its IP interfaces prior to setting the underlay address on it, apparently as a workaround for https://github.com/oxidecomputer/stlouis/issues/435. That's fine for other zones, but that races with the sled-agent setting that underlay address later in the switch zone. It's possible for the zone-setup-service to delete the interface after those addresses are set, which obviously prevents the rest of the control plane from deploying correctly. This fixes the issue by simply removing that call to ipadm delete-if in the zone-setup-service. The mentioned issue has been resolved, and the workaround is no longer needed.
• Move the zone-network-setup service depend on the network milestone, instead of multi-user. This just moves it earlier a bit in the dependency graph, though should not be strictly necessary. We might want to move the sled-agent's notion of "zone readiness" to depend on multi-user instead of single-user in the future, so this could help with that.
• Extract out a few constants, some whitespace cleanup

[bnaecker]

I've now run this a few dozen times on my machine, and not seen any of the races mentioned in the issue. Instead, tracing the ipadm calls always shows this:

bnaecker@shale : ~/omicron $ pfexec dtrace -n 'proc:::exec-success /execname == "ipadm" && strstr(curpsinfo->pr_psargs, "oxControlService0") != NULL || strstr(curpsinfo->pr_psargs, "fd00:1122:3344:101::2") != NULL/ { printf("zone %d: '%s'\n", curpsinfo->pr_zoneid, curpsinfo->pr_psargs); }'
dtrace: description 'proc:::exec-success ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  3   2857         exec_common:exec-success zone 11: /usr/sbin/ipadm create-if -t oxControlService0

  2   2857         exec_common:exec-success zone 11: /usr/sbin/ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService0

 15   2857         exec_common:exec-success zone 11: /usr/sbin/ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService0

  4   2857         exec_common:exec-success zone 11: /usr/sbin/ipadm show-addr -p -o ADDR oxControlService0/omicron6

  0   2857         exec_common:exec-success zone 11: /usr/sbin/ipadm show-addr -p -o TYPE oxControlService0/ll

 15   2857         exec_common:exec-success zone 11: /usr/sbin/ipadm create-addr -t -T addrconf oxControlService0/ll

  2   2857         exec_common:exec-success zone 11: bash -c /usr/sbin/ipadm create-addr -t -T static -a fd00:1122:3344:101::2/64 ox

  2   2857         exec_common:exec-success zone 11: /usr/sbin/ipadm create-addr -t -T static -a fd00:1122:3344:101::2/64 oxControlS

  4   2857         exec_common:exec-success zone 11: /usr/sbin/ipadm show-addr -p -o ADDR oxControlService0/omicron6

That order is not strictly guaranteed, since there is still a window of concurrency between the sled-agent and the zone-setup tool. Note that this is necessary: the zone-setup tool attempts to create a default route, which will fail until the zone has an underlay IPv6 address. That's created by the sled-agent. As dpd and tfportd are both dependent on the zone-setup service running to completion, we can't really avoid that AFAIK.

[karencfv]

Thanks for finding the bug and providing a quick fix! Left a couple of comments, but otherwise looks good to me.

I'll leave the review of the manifest dependencies to @citrus-it though. He'll have a better grasp on that than me.

[citrus-it]

A couple of nits on the manifest, and I concur with @karencfv's comment on IPV4_STATIC_ADDROBJ_NAME.