[sled agent] Provision a reservoir for Propolis

[smklein]

See:

• https://github.com/illumos/illumos-gate/blob/79492562b32b5e6bc03e14ad2b51f986335f3709/usr/src/uts/intel/io/vmm/vmm_reservoir.c#L16-L78
• https://github.com/oxidecomputer/illumos-gate/blob/stlouis/usr/src/cmd/rsrvrctl/rsrvrctl.c
  • I found this in the ramdisk image in /usr/lib/rsrvrctl

This will probably be a blocker for accurate accounting in #2648

[jordanhendricks]

Note: When oxidecomputer/propolis#410 lands, and this work lands, we can update omicron/package-manifest.toml to pull in the new propolis-server build artifact. (Without the reservoir configured, the change in oxidecomputer/propolis#410 won't go well.)

[jordanhendricks]

#3200 gets us most of the way here: sled-agent is now capable of resizing the reservoir, but it won't use it by default. There is at least one blocker for enabling the reservoir by default:

• sled-agent needs to set up a swap device at startup: Sled Agent should be capable of utilizing swap space on the M.2 dump device #2858

We can test the reservoir independently by adding swap devices manually. I will add notes to #2858 about what the swap device setup will look like.

From initial testing of reservoir use in #2684, there were potentially other blockers identified but not yet understood aiui. So I will update this ticket if we identify more issues.

[jordanhendricks]

The tickets got a little muddled here as to what is covering what, but the upshot is that with #3607 we are now provisioning the reservoir in production, which was our goal for FCS.

There is still some work to do here to manage resource limits along guest memory lines intelligently (#3228), but that is an MVP goal.