Mapping Contracts to Kubernetes CRDs

[nyarly]

A cursory review makes it seem that there's a reasonable overlap between the OpenAPI v3 Validation and CEL languages that Kubernetes uses to validate CRDs and Nickel contracts and types.

I wonder if it would be possible to write Nickel contracts and convert them into (parts of) a CRD or vice versa. In part, I'm coming from CUE's ability to import Go and JSONSchema files, and wondering if there's a sensible parallel in Nickel.

In order of decreasing preference and increasing expectation:

• Is there an existing Nickel feature to translate CRDs (or OpenAPI schema or...) into Nickel contracts? Separately, is there a feature to do the reverse?
• Is there an ecosystem project that does this? Something akin to nickel-nix.
• Is this a sensible thing to want? Is there a (near-enough) bijection from one language to the other?
• Failing that, is there a (near-enough to) one-to-one mapping from one language to the other? Can we at least e.g. model CRDs in Nickel, even if CRDs are too anemic to model Nickel (or vice versa)?

The last two bullets amount to "would it be worth the time to try to build such a tool?" An important consideration in that vein would be this: would anyone else have use for such a thing, and if so why doesn't it exist already?

[vkleen]

There has been interest in building a JSON schema to Nickel contract converter, see also #418. There is a bit of a conceptual issue with anyOf, etc., in the presence of higher order contracts like in Nickel. However, for the conversion from JSON schema we should be able to get away with a restricted version. I'm currently hacking away at a prototype 😄

[nyarly]

Thanks! That was a really interesting tour of the state of the art right now.

I still wonder if there's work (or an avenue...) the other direction: can Nickel contracts be represented as OpenAPI schemata? The lack of unions makes it clear that OpenAPI is in some way more expressive that contracts are - is the reverse also true, or is OpenAPI a superset of contracts?

Likewise, the conversion to predicate contracts seems like it wouldn't be onerous in this context, since the Kubernetes resources are going to have to be completely serialized to be useful anyway. Restricted (future) combinators for contracts would be really useful but they don't seem like a hard requirement for this application.

Related, I'm thinking along these lines: what I'm looking for is for data to be able to flow between a tool that embeds Nickel and Kubernetes. In general that will mean that a document (a K8s Resource, usually) will need to be checked against both validations. Part of the problem space looks like this:

	Passes OpenAPI validation	Fails OpenAPI validation
Passes Nickel contract	✔️	❌
Fails Nickel contract	❌	✔️

That would be the ideal - anything Nickel accepts, I can assume Kubernetes will likewise accept and vice versa. But depending on the use case, I might be willing to relax one of the ❌es - For instance, I might be happy to accept that most validation would fail in Nickel, but there's some cases that Nickel would accept but wouldn't apply in Kubernetes.

One application I imagine is defining a custom resource in Nickel, and then emitting the corresponding CRD, in which case (and granted that Nickel->OpenAPI is possible) the above chart would be satisfied, right? The subset of OpenAPI used to represent the contracts would be all that would be used in such a process.

[yannham]

There are several ways to interpret expressiveness and "what can be encoded in what" here. In fact Nickel is a Turing-complete language and contracts can be arbitrary predicate, so on paper, you can design a translation scheme from OpenAPI (or any schema formalism, for that matter) to Nickel contracts. For the same reason, I believe you can't - on paper, again - hope for a generic and complete translation from Nickel schemas to openAPI (I'm not entirely familiar with the OpenAPI spec, but I imagine you can't encode arbitrary predicates?)

The question is then what is the shape of those schemas, and what are you using schemas for. If you only care about validation, and don't care about preserving Nickel's lazyness (which, I suspect, might be the case for Kubernetes validation), then any mechanical translation would do (translate the "base" contracts as Nickel predicates, and then implement the various combinators - this is the initial approach of @vkleen for JSON schema currently, I believe). Maybe you'll have to work a bit to make the error message well localized (it's working well when you rely on Nickel builtin contract combinators, like arrays, records, etc. but to implement anyOf, oneOf, and such you'll need to have you own predicate version of arrays and records), but it's doable.

However, doing so, you'll lose another important (IMHO) aspect: in-code completion with the LSP (and showing contracts and documentation on hover). The generated contracts would basically be opaque functions, which couldn't be leveraged by the tooling except for running them and check if something satisfy the schema. The game is thus to rely on builtin contracts and contract combinators as much as possible, which is more challenging (and where the or issue comes from: you can't have both or and lazyness, and builtin contracts chose lazyness). This is our second step for JSON schema: for example, if as long as you don't encounter the first boolean operation like or or not, you can use Nickel builtin contract combinators up to there. Even for an or, you might be smart and still rely partially on builtin contracts depending on the shape of the arguments.

That being said, even a full predicate translation would be in some sense structured: the translation would use combinators as well, just not the one of the language. Thus, because translating things like anyOf and oneOf is a widespread issue enough (JSON schema, openAPI, and I imagine pretty much any kind of schema specification has an or combinator) we could consider making the LSP and the tooling in general understand those additional combinators, as if they were built in. In some sense you would have lazy but not or-able contracts and combinators (the current ones), and eager but predicate-thus-supporting-any-boolean-operation-like-not-or-or contracts and combinators, and have the LSP understand both.

Also, you might find subsets of Nickel contracts that are easily translatable to OpenAPI (obvious but correct example: the image of a translation from OpenAPI to Nickel would probably be easy to convert back to OpenAPI). Another solution is to ignore custom contracts - try to translate as much structure as you can, but when you encounter a | Strings | CustomAndStrangeStringProperty (two contract application), just use a string and add the name and documentation of CustomAndStringStringPoperty inside the documentation field of the value in the schema. It's not lossless, but if there's information that you can't representation anyway in the target formalism, this might be reasonable.

[nyarly]

One wrinkle with Nickel->OpenAPI that strikes me is predicate contracts - does nickel-lang have enough introspection into the contracts to be able to represent them in JSONSchema?

If so, is that even possible? Perhaps CEL is expressive enough to allow for general purpose Nickel contracts, but I wouldn't be surprised if it was a challenge.

I'm not sure exactly how to express the idea, but are there, perhaps, subsets of each language as-expressible-in the other? I'm thinking of something like canonical JSON. Maybe a nickel2openapi tool enforces/parses not just Nickel but a well-defined Nickel-as-expressible-in-OpenAPI. (Likewise, an openapi2nickel tool with suitable replacements).

The related issue would be: can any of the above be accomplished idiomatically? Contemplating any of this, I imagine one naive solution being to lean hard on CEL rather than trying to use more conventional features of JSONSchema, and while that would fit the bill, I can imagine Kubernetes operators rolling their eyes at "another Nickel-generated CRD."