Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release/drs 1.5.0 #408

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
12 changes: 6 additions & 6 deletions .spec-docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@
"branchPathBase": "preview",
"redocTheme": "ga4gh",
"buildPages": [
{
"apiSpecPath": "openapi/data_repository_service.openapi.yaml",
"htmlOutfile": "index.html",
"yamlOutfile": "openapi.yaml",
"jsonOutfile": "openapi.json"
},
{
"apiSpecPath": "pages/more-background-on-compact-identifiers/openapi.yaml",
"htmlOutfile": "more-background-on-compact-identifiers.html",
"yamlOutfile": "more-background-on-compact-identifiers.yaml",
"jsonOutfile": "more-background-on-compact-identifiers.json"
},
{
"apiSpecPath": "openapi/data_repository_service.openapi.yaml",
"htmlOutfile": "index.html",
"yamlOutfile": "openapi.yaml",
"jsonOutfile": "openapi.json"
}
]
}
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

# Data Repository Service (DRS) API

<sup>`develop` branch status: </sup>[![Build Status](https://app.travis-ci.com/ga4gh/data-repository-service-schemas.svg?branch=develop)](https://app.travis-ci.com/ga4gh/data-repository-service-schemas.svg?branch=develop)
<a href="https://ga4gh.github.io/data-repository-service-schemas/preview/develop/swagger.yaml"><img src="http://online.swagger.io/validator?url=https://ga4gh.github.io/data-repository-service-schemas/preview/develop/swagger.yaml" alt="Swagger Validator" height="20em" width="72em"></A> [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1405753.svg)](https://doi.org/10.5281/zenodo.1405753)
<sup>`develop` branch status: </sup>[![Build Status](https://app.travis-ci.com/ga4gh/data-repository-service-schemas.svg?branch=develop)](https://app.travis-ci.com/github/ga4gh/data-repository-service-schemas/builds)
<a href="https://editor.swagger.io/?url=https://ga4gh.github.io/data-repository-service-schemas/preview/develop/openapi.yaml"><img src="https://validator.swagger.io/validator?url=https://ga4gh.github.io/data-repository-service-schemas/preview/develop/openapi.yaml" alt="Swagger Validator" height="20em" width="72em"></A> [![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1405753.svg)](https://doi.org/10.5281/zenodo.1405753)

The [Global Alliance for Genomics and Health](http://genomicsandhealth.org/) (GA4GH) is an international coalition, formed to enable the sharing of genomic and clinical data.

Expand All @@ -26,6 +26,7 @@ For more information see our HTML documentation links in the table below.
| **Branch** | **Reference Documentation** | Swagger Editor |
| --- | --- | --- |
| **develop**: the stable development branch, into which feature branches are merged | [HTML](https://ga4gh.github.io/data-repository-service-schemas/preview/develop/docs/) | [Swagger Editor](https://editor.swagger.io?url=https://ga4gh.github.io/data-repository-service-schemas/preview/develop/openapi.yaml) |
| **release 1.5.0**: The 1.5.0 release of DRS adds fields for indicating cold storage, explict cloud locations, object count and size metadata, and best practice guidance for using GA4GH Data Connect with DRS. | [HTML](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.5.0/docs/) | [Swagger Editor](https://editor.swagger.io?url=https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.5.0/openapi.yaml) |
| **release 1.4.0**: The 1.4.0 release of DRS adds bulk operations. | [HTML](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.4.0/docs/) | [Swagger Editor](https://editor.swagger.io?url=https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.4.0/openapi.yaml) |
| **release 1.3.0**: The 1.3.0 release of DRS adds the ability to use the options request method to return the auth issuers for a specific DRS object. | [HTML](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.3.0/docs/) | [Swagger Editor](https://editor.swagger.io?url=https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.3.0/openapi.yaml) |
| **release 1.2.0**: The 1.2.0 release of DRS adds the standardized `/service-info` endpoint, and 2 `POST` endpoints that are functionally equivalent to the current `GET` endpoints, but they enable the submission of large passport tokens in the request body. | [HTML](https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.2.0/docs/) | [Swagger Editor](https://editor.swagger.io?url=https://ga4gh.github.io/data-repository-service-schemas/preview/release/drs-1.2.0/openapi.yaml) |
Expand Down
1 change: 0 additions & 1 deletion openapi/components/parameters/BulkObjectIdNoPassport.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ description: The object that contains the DRS object IDs array
properties:
bulk_object_ids:
type: array
description: DRS object IDs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor question: why was this description deleted?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was a duplicate "description" entry here which caused a silent build failure for our documentation (took a long time for me to find that!!)

items:
type: string
description: An array of ObjectIDs.
12 changes: 12 additions & 0 deletions openapi/components/schemas/AccessMethod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,23 @@ properties:
An arbitrary string to be passed to the `/access` method to get an `AccessURL`.
This string must be unique within the scope of a single object.
Note that at least one of `access_url` and `access_id` must be provided.
cloud:
type: string
description: >-
Name of the cloud service provider that the object belongs to.
If the cloud service is Amazon Web Services, Google Cloud Platform or Azure the values should be `aws`, `gcp`, or `azure` respectively.
example: aws, gcp, or azure

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how pedantic you want to be but this is more a repeat of the description and not a valid example. Maybe aws to keep it consistent with your region example?

region:
type: string
description: >-
Name of the region in the cloud service provider that the object belongs to.
example: us-east-1
available:
type: boolean
description: >-
Availablity of file in the cloud.
This label defines if this file is immediately accessible via DRS. Any delay or requirement of thawing mechanism if the file is in offline/archival storage is classified as 0 or unavailable.
example: true
authorizations:
allOf:
- $ref: './Authorizations.yaml'
Expand Down
6 changes: 6 additions & 0 deletions openapi/components/schemas/DrsService.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ properties:
maxBulkRequestLength:
type: integer
description: The max length the bullk request endpoints can handle (>= 1) before generating a 413 error e.g. how long can the arrays bulk_object_ids and bulk_object_access_ids be for this server.
objectCount:
Copy link

@davidlougheed davidlougheed Oct 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the documentation, these stats show as top-level in the "response sample", vs in a "stats" object in the example

type: integer
description: The total number of objects in this DRS service.
totalObjectSize:
type: integer
description: The total size of all objects in this DRS service in bytes. As a general best practice, file bytes are counted for each unique file and not cloud mirrors or other redundant copies.
type:
type: object
required:
Expand Down
2 changes: 1 addition & 1 deletion openapi/data_repository_service.openapi.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
openapi: 3.0.3
info:
title: Data Repository Service
version: 1.4.0
version: 1.5.0
x-logo:
url: 'https://www.ga4gh.org/wp-content/themes/ga4gh/dist/assets/svg/logos/logo-full-color.svg'
termsOfService: 'https://www.ga4gh.org/terms-and-conditions/'
Expand Down
9 changes: 7 additions & 2 deletions openapi/paths/service-info.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
get:
summary: Retrieve information about this service
description: |-
Returns information about the DRS service
Returns information about the DRS service along with stats pertaning to total object count and cumulative size in bytes.

Extends the
[v1.0.0 GA4GH Service Info specification](https://github.com/ga4gh-discovery/ga4gh-service-info)
Expand All @@ -22,9 +22,14 @@ get:
...
"type": {
"group": "org.ga4gh",
"artifact": "drs"
"artifact": "drs",
"version": "1.5"
}
...
"stats": {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does not match what the response sample indicates

additionally, I would prefer some consistency with projects like refget and htsget, where the service-specific additions to the service-info response are nested in an object named after the service artifact

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey David, could you point me to the specifics for refget and htsget. I would love to understand how they are written so we could incorporate those into DRS

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out @davidlougheed , I think it's a good idea to be consistent here so Michael or I will take a look and try to address your feedback ASAP.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For refget, see http://samtools.github.io/hts-specs/refget.html - specifically GET /sequence/service-info and the Example JSON request and response section below it. They put myriad service-specific service info properties in a refget sub-object, which I like as a pattern since it makes it clear it's an extension of the service-info spec.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I don't feel as strongly about how the stats are nested in the response as much as I feel that they should be nested, in one form or another.

"objectCount": 774560,
"totalObjectSize": 4018437188907752
}
}
```

Expand Down
2 changes: 1 addition & 1 deletion openapi/tags/ServiceRegistry.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Example listing of a DRS API registration from a service registry's `/services`
"type": {
"group": "org.ga4gh",
"artifact": "drs",
"version": "1.4.0"
"version": "1.5.0"
},
"description": "The Data Repository Service (DRS) API ...",
"organization": {
Expand Down
5 changes: 4 additions & 1 deletion pages/more-background-on-compact-identifiers/openapi.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
openapi: 3.0.3
info:
title: More Background on Compact Identifiers
version: 1.4.0
version: 1.5.0
x-logo:
url: 'https://www.ga4gh.org/wp-content/themes/ga4gh/dist/assets/svg/logos/logo-full-color.svg'
termsOfService: 'https://www.ga4gh.org/terms-and-conditions/'
Expand All @@ -27,3 +27,6 @@ tags:
- name: Example DRS Client Compact Identifier-Based URI Resolution Process - Registering a new Compact Identifier for Your DRS Server
description:
$ref: ./tags/ExampleRegisterIdentifier.md
- name: Example How To Handle Extra Metadata for DRS Objects
description:
$ref: ./tags/DRSPlusDataConnect.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
## DRS and Data Connect

With DRS objects it may be necessary to attach additional metadata to your objects. We believe that a change to the API of DRS to include metadata is not in the spirit of the DRS spec and in general DRS should have no knowledge of the metadata associated with the objects. We have found that a good GA4GH standard to support this is Data Connect (https://github.com/ga4gh-discovery/data-connect). The general approach would be to have a Data Connect service on your platform and to include "tables" with the ID matching your DRS ID for the same object. This means that if you have metadata associated with an object id `abcd` (ex. additional information about Compound Objects) all you need to do is request the information from the Data Connect client at `/tables/abcd/info`. There are optional functionalities of Data Connect, such as querying of tables, but we do not explore them or give any recommendations here.

Here is an example of using Data Connect with DRS in the fasp-scripts repository (https://github.com/ga4gh/fasp-scripts/blob/master/notebooks/drs/DRS%20File%20Data.ipynb). In this notebook we can see that data connect is used to get DRS IDs from a platform. Those DRS IDs are then used to gather aditional information about the file that might be necessary for analysis. This is just one example of how DRS and Data Connect can interact with each other to gather information about data on a platform.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use "Data Connect" here "... can see that data connect is used to ...".