[nexus] switch nexus internal API to a trait, add scaffolding to manage it

Cargo.toml

@@ -20,6 +20,7 @@ members = [
     "dev-tools/crdb-seed",
     "dev-tools/omdb",
     "dev-tools/omicron-dev",
+    "dev-tools/openapi-manager",
     "dev-tools/oxlog",
     "dev-tools/reconfigurator-cli",
     "dev-tools/releng",
@@ -46,6 +47,7 @@ members = [
     "nexus/db-model",
     "nexus/db-queries",
     "nexus/defaults",
+    "nexus/internal-api",
     "nexus/inventory",
     "nexus/macros-common",
     "nexus/metrics-producer-gc",
@@ -109,6 +111,7 @@ default-members = [
     "dev-tools/crdb-seed",
     "dev-tools/omdb",
     "dev-tools/omicron-dev",
+    "dev-tools/openapi-manager",
     "dev-tools/oxlog",
     "dev-tools/reconfigurator-cli",
     "dev-tools/releng",
@@ -138,6 +141,7 @@ default-members = [
     "nexus/db-model",
     "nexus/db-queries",
     "nexus/defaults",
+    "nexus/internal-api",
     "nexus/inventory",
     "nexus/macros-common",
     "nexus/metrics-producer-gc",
@@ -310,6 +314,7 @@ hyper = "0.14"
 hyper-rustls = "0.26.0"
 hyper-staticfile = "0.9.5"
 illumos-utils = { path = "illumos-utils" }
+indent_write = "2.2.0"
 indexmap = "2.2.6"
 indicatif = { version = "0.17.8", features = ["rayon"] }
 installinator = { path = "installinator" }
@@ -344,6 +349,7 @@ nexus-db-model = { path = "nexus/db-model" }
 nexus-db-queries = { path = "nexus/db-queries" }
 nexus-defaults = { path = "nexus/defaults" }
 nexus-inventory = { path = "nexus/inventory" }
+nexus-internal-api = { path = "nexus/internal-api" }
 nexus-macros-common = { path = "nexus/macros-common" }
 nexus-metrics-producer-gc = { path = "nexus/metrics-producer-gc" }
 nexus-networking = { path = "nexus/networking" }
@@ -449,6 +455,7 @@ shell-words = "1.1.0"
 signal-hook = "0.3"
 signal-hook-tokio = { version = "0.3", features = [ "futures-v0_3" ] }
 sigpipe = "0.1.3"
+similar = { version = "2.5.0", features = ["bytes"] }
 similar-asserts = "1.5.0"
 # Don't change sled's version on accident; sled's on-disk format is not yet
 # stable and requires manual migrations. In the limit this won't matter because

README.adoc

@@ -181,16 +181,23 @@ By default, Cargo does not operate on the tests.  Cargo's check/build/clippy com
 Each service is a Dropshot server that presents an HTTP API. The description of
 that API is serialized as an
 https://github.com/OAI/OpenAPI-Specification[OpenAPI] document which we store
-in link:./openapi[`omicron/openapi`] and check in to this repo. In order to
-ensure that changes to those APIs are made intentionally, each service contains
-a test that validates that the current API matches. This allows us 1. to catch
-accidental changes as test failures and 2. to explicitly observe API changes
-during code review (and in the git history).
+in link:./openapi[`omicron/openapi`] and check in to this repo. Checking in
+these generated files allows us:
+
+. To catch accidental changes as test failures.
+. To explicitly observe API changes during code review (and in the git history).
 
 We also use these OpenAPI documents as the source for the clients we generate
 using https://github.com/oxidecomputer/progenitor[Progenitor]. Clients are
 automatically updated when the coresponding OpenAPI document is modified.
 
+There are currently two kinds of services based on how their corresponding documents are generated: *managed* and *unmanaged*. Eventually, all services within Omicron will transition to being managed.
+
+* A *managed* service is tracked by the `cargo xtask openapi` command, using Dropshot's relatively new API trait functionality.
+* An *unmanaged* service is defined the traditional way, by gluing together a set of implementation functions, and is tracked by an independent test.
+
+To check whether your document is managed, run `cargo xtask openapi list`: it will list out all managed OpenAPI documents. If your document is not on the list, it is unmanaged.
+
 Note that Omicron contains a nominally circular dependency:
 
 * Nexus depends on the Sled Agent client
@@ -201,7 +208,13 @@ Note that Omicron contains a nominally circular dependency:
 We effectively "break" this circular dependency by virtue of the OpenAPI
 documents being checked in.
 
-In general, changes any service API **require the following set of build steps**:
+==== Updating Managed Services
+
+See the documentation in link:./dev-tools/openapi-manager[`dev-tools/openapi-manager`] for more information.
+
+==== Updating Unmanaged Services
+
+In general, changes to unmanaged service APs **require the following set of build steps**:
 
 . Make changes to the service API.
 . Update the OpenAPI document by running the relevant test with overwrite set:

dev-tools/openapi-manager/Cargo.toml

@@ -0,0 +1,25 @@
+[package]
+name = "openapi-manager"
+version = "0.1.0"
+edition = "2021"
+license = "MPL-2.0"
+
+[lints]
+workspace = true
+
+[dependencies]
+anyhow.workspace = true
+atomicwrites.workspace = true
+camino.workspace = true
+clap.workspace = true
+dropshot.workspace = true
+fs-err.workspace = true
+indent_write.workspace = true
+nexus-internal-api.workspace = true
+omicron-workspace-hack.workspace = true
+openapiv3.workspace = true
+openapi-lint.workspace = true
+owo-colors.workspace = true
+serde_json.workspace = true
+similar.workspace = true
+supports-color.workspace = true

dev-tools/openapi-manager/README.adoc

@@ -0,0 +1,103 @@
+= OpenAPI manager
+
+This tool manages the OpenAPI documents (JSON files) checked into Omicron's `openapi` directory, using Dropshot's support for *API traits*.
+
+NOTE: For more information about API traits, see https://rfd.shared.oxide.computer/rfd/0479[RFD 479].
+
+Currently, a subset of OpenAPI documents is managed by this tool. Eventually, all of the OpenAPI documents in Omicron will be managed by this tool; work to make that happen is ongoing.
+
+To check whether your document is managed, run `cargo xtask openapi list`: it will list out all managed OpenAPI documents. If your document is not on the list, it is unmanaged.
+
+== Basic usage
+
+The OpenAPI manager is meant to be invoked via `cargo xtask openapi`. Currently, three commands are provided:
+
+* `cargo xtask openapi list`: List information about currently-managed documents.
+* `cargo xtask openapi check`: Check that all of the managed documents are up-to-date.
+* `cargo xtask openapi generate`: Update and generate OpenAPI documents.
+
+There is also a test which makes sure that all managed documents are up-to-date, and tells you to run `cargo xtask openapi generate` if they aren't.
+
+=== API crates [[api_crates]]
+
+The OpenAPI manager has dependencies on a set of *API crates*. An API crate is a Rust library that consists of the API trait, and possibly supporting types. Each OpenAPI document should have a separate API crate.
+
+To keep compile times down, ensure that the API crate has as few dependencies as possible. In particular, *strongly avoid any dependencies on Diesel or other database logic*.
+
+The ideal set of dependencies is:
+
+* Common crates within omicron: `omicron-common`, perhaps `omicron-uuid-kinds` if typed UUIDs are in use, and a `types` crate for your service.
+* Core external crates: `dropshot`, `serde`, `schemars`, and `uuid`.
+
+For an archetypal way to organize code, see the dependency graph in https://rfd.shared.oxide.computer/rfd/0479#functions_vs_traits[RFD 479's _Choosing between functions and traits_].
+
+== Managing OpenAPI documents
+
+For OpenAPI documents to be managed by this tool, the corresponding interfaces must be defined via *API traits* rather than traditional Dropshot function-based servers.
+
+TIP: For examples within Omicron, search the repo for `dropshot::api_description`.
+
+=== Adding new documents
+
+If you're defining a new service fronted by OpenAPI, first create an API crate (see <<api_crates>> above).
+
+. Add the API crate to the workspace's `Cargo.toml`: `members` and `default-members`, and a reference in `[workspace.dependencies]`.
+. Following the example in https://rfd.shared.oxide.computer/rfd/0479#guide_trait_definition[RFD 479's _Trait definition_], define the API trait.
+
+In the implementation crate:
+
+. Add a dependency on the API crate.
+. Following the example in https://rfd.shared.oxide.computer/rfd/0479#guide_api_implementation[RFD 479's _API implementation_], provide an implementation of the trait.
+
+Once the API crate is defined, perform the steps in <<add_to_manager>> below.
+
+=== Converting existing documents
+
+Existing, unmanaged documents are generated via *function-based servers*: a set of functions that some code combines into a Dropshot `ApiDescription`. (There is also likely an expectorate test which ensures that the document is up-to-date.)
+
+The first step is to convert the function-based server into an API trait. To do so, create an API crate (see <<api_crates>> above).
+
+. Add the API crate to the workspace's `Cargo.toml`: `members` and `default-members`, and a reference in `[workspace.dependencies]`.
+. Follow the instructions in https://rfd.shared.oxide.computer/rfd/0479#guide_converting_functions_to_traits[RFD 479's _Converting functions to API traits_] for the API crate.
+
+In the implementation crate:
+
+. Continue following the instructions in https://rfd.shared.oxide.computer/rfd/0479#guide_converting_functions_to_traits[RFD 479's _Converting functions to API traits_] for where the endpoint functions are currently defined.
+. Find the test which currently manages the document (try searching the repo for `openapi_lint::validate`). If it performs any checks on the document beyond `openapi_lint::validate` or `openapi_lint::validate_external`, see <<extra_validation>>.
+
+Next, perform the steps in <<add_to_manager>> below.
+
+Finally, remove:
+
+. The test which used to manage the document. The OpenAPI manager includes a test that will automatically run in CI.
+. The binary subcommand (typically called `openapi`) that generated the OpenAPI document. The test was the only practical use of this subcommand.
+
+=== Adding the API crate to the manager [[add_to_manager]]
+
+Once the API crate is defined, inform the OpenAPI manager of its existence. Within this directory:
+
+. In `Cargo.toml`, add a dependency on the API crate.
+. In `src/spec.rs`, add the crate to the `all_apis` function. (Please keep the list sorted by filename.)
+
+To ensure everything works well, run `cargo xtask openapi generate`.
+
+* Your OpenAPI document should be generated on disk and listed in the output.
+* If you're converting an existing API, the only changes should be the ones you might have introduced as part of the refactor. If there are significant changes, something's gone wrong--maybe you missed an endpoint?
+
+==== Performing extra validation [[extra_validation]]
+
+By default, the OpenAPI manager does basic validation on the generated document. Some documents require extra validation steps.
+
+It's best to put extra validation next to the trait, within the API crate.
+
+. In the API crate, add dependencies on `anyhow` and `openapiv3`.
+. Define a function with signature `fn extra_validation(openapi: &openapiv3::OpenAPI) -> anyhow::Result<()>` which performs the extra validation steps.
+. In `all_apis`, set the `extra_validation` field to this function.
+
+== Design notes
+
+The OpenAPI manager uses the new support for Dropshot API traits described in https://rfd.shared.oxide.computer/rfd/0479[RFD 479].
+
+With traditional function-based Dropshot servers, generating the OpenAPI document requires the implementation to be compiled. With API traits, that is no longer necessary. The OpenAPI manager leverages this to provide a fast and easy way to regenerate API documents.
+
+This does mean that the OpenAPI manager requires the use of API traits, and that eventually all of Omicron's Dropshot APIs should be switched over to traits.