Support $dynamicAnchor and $dynamicRef

[brandonbloom]

Motivation

These keywords are defined in the JSON Schema 2020-12 dialect.

Release notes: https://json-schema.org/draft/2020-12/release-notes

These are useful for representing generics: https://json-schema.org/blog/posts/dynamicref-and-generics

Specification of dynamic scope rules: https://json-schema.org/draft/2020-12/json-schema-core#name-lexical-scope-and-dynamic-s

Proposed solution

Schemas containing $defs generate types that include the defs as members. If the schemas contain $dynamicAnchor, they create protocols with associated types instead of concrete types with type aliases. The schema of the dynamic-anchored types are used as the type constraint for the associated types.

Dynamic reference schemas (those containing the $dynamicRef keyword) refer to associated types and may supply generic parameters using their own $defs and $dynamicAnchor keywords.

Alternatives considered

Avoid generating protocols / associated types, and instead always generate concrete types, treating dynamic refs/anchors as template instantiation.

Additional information

This would depend on support in OpenAPIKit. Upstream issue filed: mattpolzin/OpenAPIKit#359

[czechboy0]

Thanks @brandonbloom - I'll mark this one as blocked, until the OpenAPIKit support is landed.

[czechboy0]

I wonder, have you also considered generating generic types, instead of protocols? From my reading of the specification, that might be the closest representation in Swift to the requested feature.

[mattpolzin]

I've begun exploring this in OpenAPIKit. I've only thought on it for a few minutes so far, but I wanted to make sure to extend an invitation for help deciding how to integrate these new references into OpenAPKit if anyone over here has thoughts; I know it isn't always possible to comment directly on third party repositories, I will do my best to coordinate any conversation that happens split between the two disparate GitHub issues.

My initial thinking is I cannot simply support an additional key and boolean property on the JSONReference type since that type is deeply integrated with knowledge about how $ref tags behave and $dynamic-ref tags have different semantics and rules -- even that assumption is up for discussion, though.

[czechboy0]

I wonder what the minimum support for a generic type would be here. Basically, knowing that a type is generic over N types, and OpenAPIKit filling them in.

Re references, in 3.1 there's OpenAPI.Reference, maybe that's where some of it could be added?

[mattpolzin]

OpenAPI.Reference may not quite fit the needs here since that type wraps a reference and adds description/summary overriding; it does manage to introduce different behavior compared to JSONReference, but since it was introduced to support the OpenAPI-specific extensions to $ref it isn't currently supported within JSONSchema (it's used everywhere outside of JSONSchema, though).

[czechboy0]

Fair enough - yeah maybe a way to make progress here would be to first identify the minimum work needed to support the simple List<T> use case, and generalize from there.

[mattpolzin]

@brandonbloom do you have an example or two of OpenAPI documents that take advantage of $dynamicRef? I'm curious if they tend to use local $defs within the json-schemas or if (being OpenAPI Documents) they tend to put the $dynamicAnchors inside the #components/schemas which would be my instinct personally.

[brandonbloom]

I wonder, have you also considered generating generic types, instead of protocols? From my reading of the specification, that might be the closest representation in Swift to the requested feature.

I considered it briefly & perhaps discarded it prematurely. The reasons I was immediately drawn to protocols is because generics use positional type parameters, which don't have a clean mapping to the dynamicRef concept as far as I can tell. Associatedtypes seem more similar to what the local $defs look like.

do you have an example or two

Examples from Github: https://github.com/search?q=%28%22%24dynamicRef%22+OR+%22%24dynamicAnchor%22%29+%28path%3A**%2F*.yaml+OR+path%3A**%2F*.yml+OR+path%3A**%2F*.json%29&type=code

[mattpolzin]

I was moreso curious if you had examples that motivated you to request the feature; if it turns out that OpenAPIKit support for dynamic reference use-cases does not come all at once, I'd like to prioritize support for those situations that are most pressing for consumers of OpenAPI documents (as opposed to json-schema more broadly). It's fine if not.

[mattpolzin]

maybe a way to make progress here would be to first identify the minimum work needed to support the simple List<T> use case, and generalize from there.

In that vein, I have begun to brainstorm what exactly OpenAPIKit tells a consumer beyond "here's a dynamic reference." That's the bare minimum (and net addition to OpenAPIKit): Parsing $dynamicRef, $dynamicAnchor, and $defs. Beyond that, when OpenAPIKit hits a $dynamicRef, that might be something the Swift generator wants to turn into a protocol or generic type, but since OpenAPIKit uses value types to represent JSONSchema, it cannot use a reference to represent an infinitely recursive structure (like a tree or a list of lists).

That leads me to the question of what non-referential representation would be most helpful to the Swift generator. One potential answer is that OpenAPIKit could surface the $id of the dynamic anchor that a given $dynamicRef points to (this assumes the anchored schema has an $id) -- that would save the Swift generator from needing to crawl the schema to calculate the dynamic context itself. Maybe that's enough for the Swift generator to then create a type declaration for the anchor schema and store it under its $id to then later refer to that type when hitting a $dynamicRef that OpenAPIKit says resolves to the given $id.

In reality, dynamic anchors do not always have $ids, so if the above strategy makes sense even superficially, we'd need to come up with the desirable canonical way of identifying schemas (possibly even an algorithm where $id is preferred and the key of an entry in $defs is a fallback, etc.).

---

It's possible this idea is flawed because there's no inherent need to uniquely and globally identify dynamically referenced schemas: if you are actively crawling the json-schema and tracking the dynamic context, you don't need globally unique you just need to look at the current local dynamic context. Another idea is OpenAPIKit could surface that dynamic context as easily accessible for any given $dynamicAnchor, though my gut says that doing so would necessitate some form of cycle detection and avoidance.

[brandonbloom]

I was moreso curious if you had examples that motivated you to request the feature

I'm using it in a schema I'm developing. The simplest example is that I have a generic-type for a pagination container. In TypeScript notation:

type Page<Item> = {
  items: Item[];
  next: string | null;
}

This is then parameterized by each type of resource I'm returning, something like:

type FriendPage = Page<Friend>;

I already generate my schemas programmatically, so I'm working around the lack of support for this feature by treating generic types like macros and expanding them template-programming style. Since my schema generation is already parameterized, I can emit dynamic refs/anchors for validators that support them and then erase the generic types and emit specialized schemas as needed for other output targets.

bare minimum (and net addition to OpenAPIKit): Parsing

Although it also does not yet support dynamic refs/anchors, a newer Go library for OpenAPI specs has a two-level model: https://pb33f.io/libopenapi/model/ -- This roughly corresponds to a concrete parse tree and an abstract document object model. I'm not sure if that maps to your design at all, but that seems natural to me.

[czechboy0]

That's good context, @brandonbloom - let's focus on this simple example first: generic pages. @mattpolzin what's the first version of your approach you could see supporting this? I'd prefer us to build this incrementally, based on real-world needs.

[mattpolzin]

I've got work ahead of me just to properly represent $dynamicRef in the AST of OpenAPIKit, though nothing about that will be overly complex. But beyond the first step of being able to actually parse/encode/expose $dynamicAnchor and $dynamicRef, the next step for me will be to offer a function that under the hood crawls the AST looking for a $dynamicAnchor matching a given $dynamicRef. This will allow me to offer to downstream libraries some function that says "what is the resolved state of this $dynamicRef I am looking at right here?" I'll be able to answer e.g. Friend. Before going further (e.g. offering the ability to ask OpenAPIKit "what are all the possible specializations of Item in Page<Item>?"), I think we already need to address the question of what to do about recursion. We may not need to handle cycles to handle Page<Friend>, but if we design this feature around that subset of use-cases we may for example decide that Friend can be a fully resolved JSONSchema which will then mean we need to break that contract as soon as we want to handle recursion later.

This is what lead me to the questions I posed above in #547 (comment).

[czechboy0]

FWIW, we fully support recursive schemas in Swift OpenAPI Generator today, so I'd like for that to continue even with this feature. You're right that we should consider it now, instead of fully specializing on the example use case.

[mattpolzin]

It may help me to get the 10,000 ft view of how the generator handles other existing recursive cases it already handles -- for one thing, to help me understand where division of labor will happen for this new feature, so I am not trying to solve problems that the Swift Generator will want to write its own code for anyway.

Would you be able to link me to the relevant code and possibly give me any description you think might help me grok it quickly? I don't have time to become an expert in the whole generator codebase, so trying to be pragmatic 😄

[czechboy0]

Sure, please check this out first, and I can answer any followups: https://swiftpackageindex.com/apple/swift-openapi-generator/1.2.1/documentation/swift-openapi-generator/supporting-recursive-types

[mattpolzin]

Thanks. Even easier than reading the code! So, I am thinking that the easiest path forward is possibly for OpenAPIKit to facilitate the swift generator using roughly the same code (at the very least the same strategy) as it uses today to handle recursion via Components lookup to also handle dynamic references.

Dynamic references may also lead to recursion just like regular references, so the difference of course is that dynamic references change depending on where in the schema they live. So, today OpenAPIKit offers a lookup function on Components that allows resolving non-dynamic local references. Newly needed is a lookup function that handles dynamic refs. That's already an interesting problem to consider (looking forward to thinking on it) because OpenAPI documents kind of have two different potential contexts to consider: there's the immediate json-schema context and its $defs (starting wherever the nearest parent in the AST goes from being an OpenAPI object to a json-schema) but there's also the Components object's schemas that I think might reasonably be considered part of the dynamic context for every single json-schema in the whole document.

I think this is where I want to get to first in OpenAPIKit: parse/serialize/export dynamic refs and anchors and expose a lookup function that behaves much like the current one but understands dynamic context and can find anchors as well as local paths. There's no small chunk of work there because in addition to the dynamic context being new we've previously not needed to handle $defs or local path (outside of Components Object) lookup.

While I am working on that, which feels like important foundation, we can continue to discuss what gaps are left RE the swift generator's eventual goals. Maybe that's enough for the swift generator to handle the rest, or maybe OpenAPIKit also exposes some form of "give me all the ways this ref could resolve in the document" function a la the question of what possible specializations a generic type can have.

[czechboy0]

This sounds in line with what I was thinking, @mattpolzin. I'm happy to wait for the foundational pieces to land in OpenAPIKit, and then try to integrate them in the generator, and if we feel there are additional conveniences needed, we can take it from there.

[czechboy0]

FYI, there might be further changes in the upcoming OpenAPI v4: https://www.openapis.org/blog/2023/12/06/openapi-moonwalk-2024