Update to OpenAPI 3.0

[sarpera]

Current DRS schema definition is based on OpenAPI (fka swagger) 2.0. Shall we update it to conform to OpenAPI 3.0 (the latest version of the OpenAPI Specification)?

Immediate benefit of the update is to have new data model capabilities which would make the ongoing efforts on creating PRs a bliss.

[ddietterich]

I agree. It would be good to use 3.0.

The only downside I know is that swagger-codegen does not yet support 3.0. Anyone planning to use that tool?

[denis-yuen]

We do, but we've had successful experiments with swagger-codegen 3.0.4 https://github.com/swagger-api/swagger-codegen/releases/tag/v3.0.4 which does support OpenAPi 3.0.

[geoffjentry]

To echo - the conventional wisdom in the cloud workstream has been what @ddietterich stated, but is obviously going to be a moving target. My $0.02 is that as long as the infrastructural challenges are surmountable that we should go the 3.0 route

[briandoconnor]

What impact will this have on build? If this update just works then lets go for it. If not, then we might delay this for v0.1

[jaeddy]

Just noticed that swagger2markup (which we use to autogen documentation during the build process) doesn't currently support OpenAPI 3 — there might be some potential workarounds we could try:
Swagger2Markup/swagger2markup#340

I think we'd need to adjust the Swagger UI HTML template as well, but that should be a quick fix.

[natanlao]

If it would ease the transition, OpenAPI 3.0 and SmartAPI schema definitions can be generated from the Swagger schema by running make schemas in the project root, which will create openapi/data_repository_service.openapi.yaml and openapi/data_repository_service.smartapi.yaml.

[denis-yuen]

FYI @garyluu

[sarpera]

I’ve tried a few things and have some question:

First of all, seems like the instructions are out of date here: https://github.com/ga4gh/data-repository-service-schemas/blob/develop/README.md or
https://ga4gh.github.io/data-repository-service-schemas/#getting-started

After installing dependencies, I managed to run ga4gh_drs_server but seems like the example paths need to be updated (they refer to do in the paths).

When the example paths are manually adjusted based on the current schema paths on the develop branch, it seems to be working:
http://localhost:8080/ga4gh/drs/v1/service-info

Also, this seems to be broken:
https://data-repository-service.readthedocs.io/en/latest/

About the schema definitions, I managed to generate openapi/data_repository_service.openapi.yaml and openapi/data_repository_service.smartapi.yaml via running make schemas, using the latest version of swagger2openapi that supports OpenAPI v3.

Now about the rest of the tooling, how should we proceed?

I took a look at the some of the existing docs in the repo (https://github.com/ga4gh/data-repository-service-schemas/blob/develop/docs/html5/index.html), but they seem to be for WES. Are those just placeholders?

Also, seems like swagger2markup is not v3.0 compatible yet. Do we consider alternatives? There is a list of v3.* supported tools listed here: https://openapi.tools/

[natanlao]

@sarpera Hi! I can answer a few of these questions.

After installing dependencies, I managed to run ga4gh_drs_server but seems like the example paths need to be updated (they refer to do in the paths).

A good amount of rename work is ongoing, and some parts of the codebase (including the examples) have not been updated to reflect the name change. This work is being tracked in #206

Also, this seems to be broken:
https://data-repository-service.readthedocs.io/en/latest/

It looks like this was renamed in #193 as part of the DOS -> DRS rename. Docs are still up at the old site here. I've opened #232 to reconcile this difference.

I took a look at the some of the existing docs in the repo (https://github.com/ga4gh/data-repository-service-schemas/blob/develop/docs/html5/index.html), but they seem to be for WES. Are those just placeholders?

The new build system was built heavily on the system @jaeddy built for ga4gh/workflow-execution-service-schemas, so there are some lingering references - it looks like the file you pointed out is a build artifact from before that page was updated for this repo and can be safely removed.

[sarpera]

Thanks @natanlao!

I'll then be in hold until the issues you mentioned are resolved. Was hoping to go for somehow making openapi/data_repository_service.openapi.yaml the canonical version after making sure the rest of the tooling is working.

[brainstorm]

@denis-yuen I reckon that for codegen you could use https://github.com/OpenAPITools/openapi-generator instead of swagger-codegen 3.x. The former seems more supported/stable (for OpenAPI 3) than swagger tooling in my experience generating code for the htsget-spec.

[jaeddy]

@sarpera, @natanlao — I mentioned OpenAPI 3.0 / issues related to the documentation build system in the WES update I gave on today's Cloud WS call:
https://docs.google.com/presentation/d/1E4963kVGqS4BoYjW71u23oDLuez5te92-mXK2VMm-ao/edit?usp=sharing

From the list of v3-supported tools you linked above, I think ReDoc and widdershins both look promising. However, there are a couple things we'll need to verify before switching:

• Support for additional/custom non-spec docs: swagger2markup integrates text sections from asciidoc files (e.g., notes about the work stream, security, contact info, etc.) with those docs generated automatically from the swagger yaml). It's not immediately clear to me whether this is possible with either of the two tools.
• Export as PDF (in addition to the nicely rendered HTML UI) — again to satisfy GA4GH requirements

Finally, both tools are implemented as Node.js packages (whereas swagger2markup uses a gradle plug-in in our build setup). This should hopefully be less of a problem — just something we'd need to test and debug.

[dglazer]

@jaeddy -- thanks for adding those notes and pointers.

@sarpera -- the feeling on today's call was that while we'd all like OpenAPI 3, we need our autobuilds to work, we're leery of getting stuck halfway through a tools switch, and we're eager to make progress on defining the new methods. I'll add a comment in #213 about next steps.

[jimmyhli]

the maintainer of swagger-codegen said he won't have the time or resources to upgrade swagger-codegen to openAPI V3.0

[matthdsm]

Hi all,

Any updates on this front?

Thanks
M

[dglazer]

@jaeddy was working on it; I haven't asked for a status update recently. James?

[denis-yuen]

@jaeddy was working on it; I haven't asked for a status update recently. James?

My understanding is that James is working on https://github.com/ga4gh/gh-openapi-docs which will work on openapi3 sans swagger2markup

[denis-yuen]

the maintainer of swagger-codegen said he won't have the time or resources to upgrade swagger-codegen to openAPI V3.0

To elaborate for others, swagger2markup is what will not be updated.
swagger-codegen does support openapi3
https://github.com/swagger-api/swagger-codegen#swagger-codegen-3x-300-branch

[jaeddy]

Getting stuff working with the new docs build is on my todo list for this weekend — aiming to have updates to share on the call Monday.

[briandoconnor]

I started to look at this again as part of the larger doc build system updates. See the following sources of info on this:

• https://ga4gh.github.io/workflow-execution-service-schemas/docs/
• https://github.com/ga4gh/workflow-execution-service-schemas/tree/reorganize-spec-content
• https://github.com/ga4gh/workflow-execution-service-schemas/tree/reorganize-spec-content/openapi
• https://github.com/ga4gh/gh-openapi-docs/blob/master/docs/ga4gh-cloud-migration.md
• https://github.com/ga4gh/tool-registry-service-schemas/pull/141/files
• https://github.com/ga4gh/workflow-execution-service-schemas/blob/master/openapi/workflow_execution_service.openapi.yaml
• https://github.com/ga4gh/workflow-execution-service-schemas/compare/reorganize-spec-content
• https://github.com/ga4gh/gh-openapi-docs/blob/master/src/lib/config.js#L26

Known issues:

• branch naming convention for hub flow… need to test
• better approach needed for relative images
• might be a little difficult for includes of other extended docs

[briandoconnor]

Working in feature/issue-230-open-api-and-doc-build

So I was able to get something to build:

https://ga4gh.github.io/data-repository-service-schemas/preview/feature/issue-230-open-api-and-doc-build/docs-new/index.html

Next steps

• convert to OpenAPI 3, see https://stackoverflow.com/questions/59749513/how-to-convert-openapi-2-0-to-openapi-3-0
• convert docs to markdown and embed in the OpenAPI yaml

[uniqueg]

What's the status here? Seems like there's OpenAPI 3.0 now in WES v0.1.0, TES v0.5.0 and TRS v2.0.1. Would be nice to have this for DRS as well, so that our tool chains work more consistently.

There are also two PRs in for OpenAPI 3 bumps, #304 (adding, not replacing; DRS v0.1.0) & #327 (replacing), but no comments/feedback by maintainers...

[briandoconnor]

@uniqueg , I haven't worked on this since the summer. Do you have bandwidth? Would you like to pick it up? I'm looking for a champion for this.

[uniqueg]

@briandoconnor: Given that it has taken me more than a month to even manage to reply to this, I'm afraid I don't have the bandwidth right now. However, if it's still open when I have some more development time, I'll be happy to look at this.

[jb-adams]

@briandoconnor can I pick this one up? I've worked on the OpenAPI for htsget. I'm also familiar with @jaeddy 's docs build tool

[jb-adams]

Just merged #359 , which migrates the spec to OpenAPI. Closing this issue

@briandoconnor