omicron-package: allow building specific packages and their dependencies

[iliana]

This change allows building a specific set of packages, such as the packages that are installed directly in the host OS image (as opposed to the zones that are installed separately).

$ cargo run --release --bin omicron-package -- package --only omicron-sled-agent
    Finished release [optimized] target(s) in 0.99s
     Running `target/release/omicron-package package --only omicron-sled-agent`
Logging to: /home/iliana/git/omicron/out/LOG
    Finished release [optimized] target(s) in 0.94s
[ 0s] ##############################       8/8       omicron-sled-agent: done

The output doesn't make it clear, but a temporary println! debug statement demonstrates that this only built the necessary package:

Command { std: "cargo" "build" "-p" "omicron-sled-agent" "--features" "image-standard machine-gimlet rack-topology-multi-sled switch-asic " "--release", kill_on_drop: false }

This is to unlock some speed improvements to cargo xtask releng so that the Nexus binary can be built at the same time as the host and recovery OS images. (In practice this is more useful for local rebuilds; at present the releng CI time is reduced by about 90 seconds, although reordering the Nexus build to occur simultaneously with OS image builds has revealed other wasted time.) The actual releng changes will be in another PR along with some other improvements.

A more realistic invocation, with all the packages we need for the host OS image:

$ cargo run --release --bin omicron-package -- package --only mg-ddm-gz --only omicron-sled-agent \
>    --only overlay --only oxlog --only propolis-server --only pumpkind-gz --only switch-asic
    Finished release [optimized] target(s) in 1.02s
     Running `target/release/omicron-package package --only mg-ddm-gz --only omicron-sled-agent --only overlay --only oxlog --only propolis-server --only pumpkind-gz --only switch-asic`
Logging to: /home/iliana/git/omicron/out/LOG
    Finished release [optimized] target(s) in 1.06s
[ 0s] ##############################       1/1       pumpkind: done
[ 0s] ##############################       3/3       oxlog: done
[ 1s] ##############################       1/1       mgd: done
[ 1s] ##############################       1/1       mg-ddm-gz: done
[ 0s] ##############################      10/10      omicron-gateway-asic-customizations: done
[ 0s] ##############################      12/12      wicket: done
[ 1s] ##############################       1/1       propolis-server: done
[ 0s] ##############################      64/64      switch_zone_setup: done
[ 1s] ##############################       1/1       mg-ddm: done
[ 0s] ##############################       8/8       omicron-gateway: done
[ 0s] ##############################       1/1       lldp: done
[ 0s] ##############################      11/11      logadm: done
[ 0s] ##############################       7/7       xcvradm: done
[ 0s] ##############################      16/16      wicketd: done
[ 1s] ##############################       1/1       pumpkind-gz: done
[ 0s] ##############################       8/8       omicron-omdb: done
[ 0s] ##############################       8/8       profile: done
[ 0s] ##############################       8/8       omicron-sled-agent: done
[ 1s] ##############################       1/1       dendrite-asic: done
[ 0s] ##############################       4/4       overlay: done
[ 0s] ##############################       4/4       omicron-gateway-asic: done
[ 0s] ##############################      13/13      switch-asic: done

This is not the best experience at the moment; if any of the target-specific feature flags are not provided by any of the packages selected, Cargo complains:

$ cargo run --release --bin omicron-package -- package --only switch-asic
    Finished release [optimized] target(s) in 0.98s
     Running `target/release/omicron-package package --only switch-asic`
Logging to: /home/iliana/git/omicron/out/LOG
error: none of the selected packages contains these features: image-standard, machine-gimlet, rack-topology-multi-sled, switch-asic
Error: Failed to build packages

We could try to use cargo-metadata to go through and see if any of the enabled packages have those features, or go add the requisite feature flags to the rest of the crates in the tree built by omicron-package. (Or add them to omicron-workspace-hack???)

This is sort of an early revision of this PR; the actual implementation of selecting particular trees from a PackageMap maybe belongs in omicron-zone-package, but I don't have a good vibe on whether that's the case.

[iliana]

I ended up rewriting the dependency walker to avoid walking each node more than once.

I also ran into an issue while testing this -- Cargo absolutely refuses to run if the requested features in --features aren't present in one of the requested packages. I tried a few workarounds but ultimately settled on filtering the feature list based on output from the cargo_metadata crate.

While doing that refactor, I decided to change it to run cargo build --bin [binaries..] instead of cargo build -p [packages..], explicitly requesting the targets that omicron-package actually wants.

[smklein]

Dumb question, but when would this practically trigger? Mechanically, I understand that it's "a binary target exists in the set of all targets multiple times", but would this be like, multiple versions of one binary? How could that happen?

(Mostly asking for the sake of whoever sees this error in the wild - I'm also interested in whatever hint we could provide for "how to fix this").

[iliana]

Two binaries with the same name in the workspace (e.g. some package has a binary named "schema-updater" along with the omicron-nexus package) — while self-reviewing the code I tested to see if Cargo dealt with that, and it... doesn't! (yet, at least.)

I can see how someone might do this accidentally, and if so it might get really hard to debug.

[smklein]

nitty nitpick: This could probably be stated as:

if self.only.is_empty() {
  return packages;
}
... // Contents of "else" block

To reduce some indentation

[smklein]

To clarify, are all these features getting removed because they aren't used anywhere?

(If so, great, awesome to see these cleaned up)

[iliana]

Yep, aiui these were all in sled-agent because the features had to be defined in some package for the cargo build to work. But if you're not building sled-agent, you'll often get a failure to build because of features that don't exist in any of the selected packages.

I grepped for the feature names across the tree to identify which packages were actually using which feature names.