From 9b1f71e3f574a3b4916f9cd349b2bd1fa95e8c3e Mon Sep 17 00:00:00 2001 From: Brian Date: Mon, 29 Jul 2019 15:53:46 +0000 Subject: [PATCH 1/3] Bundle Enumeration Draft --- rfcs/text/0000-dss-bundle-enumeration.md | 43 ++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 rfcs/text/0000-dss-bundle-enumeration.md diff --git a/rfcs/text/0000-dss-bundle-enumeration.md b/rfcs/text/0000-dss-bundle-enumeration.md new file mode 100644 index 00000000..3ba56c45 --- /dev/null +++ b/rfcs/text/0000-dss-bundle-enumeration.md @@ -0,0 +1,43 @@ +### DCP PR: + +***Leave this blank until the RFC is approved** then the **Author(s)** must create a link between the assigned RFC number and this pull request in the format:* + +`[dcp-community/rfc#](https://github.com/HumanCellAtlas/dcp-community/pull/)` + +# RFC: Bundle Enumeration of the DCP Data Store (DSS) + +## Summary + +This RFC proposes bundle enumeration endpoint(s) for the DSS. + +## Author(s) + +* [Brian Hannafious](mailto:bhannafi@ucsc.edu) + +## Shepherd +***Leave this blank.** This role is assigned by DCP PM to guide the **Author(s)** through the RFC process.* + +*Recommended format for Shepherds:* + + `[Name](mailto:username@example.com)` + +## Motivation + +Currently, the only means of listing bundles stored in the DSS is the internal Elasticsearch (ES) metadata index, which +must be kept current with object storage. The DSS should provide bundle enumeration independently of the ES metadata index, +emphasizing consistency and scalability. + +### User Stories + +* As a downstream service developer, I would like to enumerate the bundle contents of the DSS so I can create my own + index. + +* As a downstream service developer, I would like to check if my index contains all the bundles in the DSS. + +## Detailed Design + +A new bundle enumeration endpoint, `GET /bundles`, will be introduced, taking replica and prefix parameters. These +parameters will be used to return a paginated listing of bundles directly from object storage. Pagination semantics +and all other semantics of this route will be in line with the established conventions of the DSS API. + +### Unresolved Questions From aebf3d380798e995450d8cbb7f5d9cf9bca7be2e Mon Sep 17 00:00:00 2001 From: Brian Date: Tue, 10 Sep 2019 16:20:30 +0000 Subject: [PATCH 2/3] prefix explained --- rfcs/text/0000-dss-bundle-enumeration.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/rfcs/text/0000-dss-bundle-enumeration.md b/rfcs/text/0000-dss-bundle-enumeration.md index 3ba56c45..8a6c62b2 100644 --- a/rfcs/text/0000-dss-bundle-enumeration.md +++ b/rfcs/text/0000-dss-bundle-enumeration.md @@ -36,8 +36,9 @@ emphasizing consistency and scalability. ## Detailed Design -A new bundle enumeration endpoint, `GET /bundles`, will be introduced, taking replica and prefix parameters. These -parameters will be used to return a paginated listing of bundles directly from object storage. Pagination semantics -and all other semantics of this route will be in line with the established conventions of the DSS API. +A new bundle enumeration endpoint, `GET /bundles`, will be introduced, taking replica and prefix parameters. A listing +all all bundles having UUIDs beginning with `prefix` will be returned directly from object storage. The prefix +parameter is intended to facilitate parallelized listings. Pagination semantics and all other semantics of this route +will be in line with the established conventions of the DSS API. ### Unresolved Questions From 18ddfca268b66178a6165b2940084877dd9b5dc9 Mon Sep 17 00:00:00 2001 From: Brian Date: Wed, 18 Sep 2019 20:26:47 +0000 Subject: [PATCH 3/3] address comments --- rfcs/text/0000-dss-bundle-enumeration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rfcs/text/0000-dss-bundle-enumeration.md b/rfcs/text/0000-dss-bundle-enumeration.md index 8a6c62b2..3d9ff464 100644 --- a/rfcs/text/0000-dss-bundle-enumeration.md +++ b/rfcs/text/0000-dss-bundle-enumeration.md @@ -36,8 +36,8 @@ emphasizing consistency and scalability. ## Detailed Design -A new bundle enumeration endpoint, `GET /bundles`, will be introduced, taking replica and prefix parameters. A listing -all all bundles having UUIDs beginning with `prefix` will be returned directly from object storage. The prefix +A new bundle enumeration endpoint, `GET /bundles/all`, will be introduced, taking replica and uuid prefix parameters. A +listing of all bundles having UUIDs beginning with `prefix` will be returned directly from object storage. The prefix parameter is intended to facilitate parallelized listings. Pagination semantics and all other semantics of this route will be in line with the established conventions of the DSS API.