RFC: HCA DCP Bundle Types and Definitions #93

malloryfreeberg · 2019-07-29T15:32:33Z

Please note: the contents of #86 were merged into this PR; please refer to the discussion in that PR for additional detail and comments addressed from reviewers.

kislyuk · 2019-07-30T21:09:44Z

rfcs/text/0000-bundle-definition.md

+|:-|:-|:-|:-|
+| Primary bundle | A bundle that contains primary (raw) data files and all related metadata files. | computational biologists; Data Processing Pipelines; Data Browser; Query Service | 1 |
+| Secondary bundle | A bundle that contains secondary (alignment, expression) data files produced by the Data Processing Pipelines, input primary data files, and all related metadata files. | computational biologists; Data Browser; Query Service; Matrix Service | 2, 3, 4(?) |
+| Tertiary bundle* | A bundle that contains tertiary (expression) data files produced by the Matrix Service, input primary and secondary data files, and all related metadata files. | computational biologists; biologists; Data Browser; Query Service | 3, 4 |


I think "tertiary (expression)" is unnecessarily confusing. Tertiary analysis can be any meta-analysis, it should not be constrained to be "expression", which itself is a confusing shorthand for "expression matrix".

I propose renaming:

"Secondary bundle" -> "Analysis bundle"

"Tertiary bundle" -> "Expression Matrix bundle"

I'm ok with renaming secondary and tertiary (in fact, that's the point of this RFC!). If we are renaming those to be nouns, we should also rename the primary bundle, perhaps:

"Primary bundle" -> "Raw (or Primary) data bundle"

"Secondary bundle" -> "Analysis bundle"

"Tertiary bundle" -> "Expression Matrix bundle"

To clarify, "Secondary/Analysis bundles" also contain expression data (gene x cell count matrices) in addition to alignment results, so I wonder if it's confusing to have these results not in the "Expression Matrix bundle".

"Raw" doesn't seem very meaningful. Someone looking at gene expression (most likely the majority of users, might think of unnormalized counts as "raw".

"Matrix" is a type of data structured, so it seems odd to include it in the name. Kind of like calling sequence experiments "fastaq bundles". How about "gene expression bundle"

Analysis doesn't have much meaning either. I was kind of thinking gene expression is the result of analysis.

Gene expression matrices are a result of both pipelines and the matrix service (pipelines make them, matrix service acts on them for users).

kislyuk · 2019-07-30T21:10:51Z

rfcs/text/0000-bundle-definition.md

+| Tertiary bundle* | A bundle that contains tertiary (expression) data files produced by the Matrix Service, input primary and secondary data files, and all related metadata files. | computational biologists; biologists; Data Browser; Query Service | 3, 4 |
+| Project bundle* | A bundle that contains project-level metadata files. This bundle type does not contain data files or metadata files related to biomaterials, protocols, processes, or files. | computational biologists; biologists; | ? |
+| Resource/Reference bundle* | A bundle that contains reference data files used by the Data Processing Pipelines(?) and all related metadata files. This bundle type does not contain data files or metadata files related to biomaterials, protocols, processes(, or projects?). | computational biologists; Data Processing Pipelines; | 2, 3, 5 |
+| DAPS bundle* | ? | ? | ? |


I think there should be a link here to #88, indicating that the term and concept of a DAPS bundle is subject to discussion.

Good point. I have added a reference.

samanehsan · 2019-09-06T20:00:46Z

rfcs/text/0000-bundle-definition.md

+
+*Acceptance criteria are the conditions that a RFC must satisfy to be accepted by users or other stakeholders.* 
+
+1. Data consumers can make use of data from the DCP without knowing what a bundle is.


I see this RFC is still in draft status, but I wanted to note here that this AC doesn't seem to fit in with the rest of the document, which focuses on defining bundle types rather than hiding bundle details from data consumers. Maybe it can be re-worded or removed?

Thank you @samanehsan, I agree it's not addressed by the RFC in its current form. I'll remove it.

Seems to me the AC above is a valid constraint on the solution, that it does not further bake in the bundle concept to the DCP interface. Users care about releases, the files in them, and the processes used to create the files.

We should be careful not to force hundereds/thousands of users to also learn our "bundle types" when we already have metadata describing the processes that creates the files.

@NoopDog That may be true, but the content of this RFC as it stands does not change the status quo in that respect nor pass judgment on how data is presented to the user.

More generally, any advocate for removing bundles from the presentation layer should come up with a concrete, actionable proposal (perhaps in another RFC) for what to replace them with.

NoopDog · 2019-09-11T16:29:55Z

It may be more useful to tag bundles by the process that created them rather than the type of data they contain. The type of data can be inferred from the process and the process is more descriptive.

I have a few more notes about this here:

#86 (comment)

kislyuk · 2019-09-11T17:31:33Z

@NoopDog how would this tagging look like in practice, and what would a subscription query look like to specifically match such bundles?

rfcs/text/0000-bundle-definition.md

TimothyTickle · 2019-09-14T01:01:24Z

rfcs/text/0000-bundle-definition.md

+| Project bundle* | A bundle that contains project-level metadata files. This bundle type does not contain data files or metadata files related to biomaterials, protocols, processes, or files. | computational biologists; biologists; | ? |
+| Resource/Reference bundle* | A bundle that contains reference data files used by the Data Processing Pipelines(?) and all related metadata files. This bundle type does not contain data files or metadata files related to biomaterials, protocols, processes(, or projects?). | computational biologists; Data Processing Pipelines; | 2, 3, 5 |
+| DAPS bundle* | Definition discussion ongoing in [PR #88](https://github.com/HumanCellAtlas/dcp-community/pull/88). | ? | ? |
+


I am not sure why the above bundles were defined in this manner other than this is de facto, but I am not sure that is the intent of this RFC, or at least does not feel like a formal understanding of bundles but more of a documentation of what we are using (without comment or systematic definitions).

@TimothyTickle I believe that is the intent. I don't think we need to "formally understand bundles". We need to define bundle types and annotate them with enough information for application developers to easily reason about them, so they can develop applications and present data to users with reasonable certainty.

TimothyTickle · 2019-09-14T01:04:19Z

rfcs/text/0000-bundle-definition.md

+Describe here...
+
+- Will contain project-level metadata only. 
+- Unclear how this bundle will be used by data consumers.


How about internal users?

Thanks, great point. Added "and DCP services" to the above.

TimothyTickle

I like the way this RFC is heading and the kinds of questions being asked here but I do not think this RFC answers those questions or is even complete.

Happy to talk if you feel I can be helpful.

TimothyTickle · 2019-09-14T01:08:43Z

rfcs/text/0000-bundle-definition.md

+
+> Asterisk (*) indicates bundles types that do not currently exist in the DCP but have been discussed as potential future bundle types to support data consumers.
+
+### New bundle types and their definitions


These new bundle types and definitions are not complete but I would love to review them when they are. Feel free to reach out.

kislyuk · 2019-09-16T17:20:48Z

Please note: I have merged the contents of #86 into this PR; please refer to the discussion in that PR for additional detail and comments addressed from reviewers.

rfcs/text/0000-bundle-definition.md

diekhans · 2019-09-16T19:42:22Z

I create issue #114 to get media types turned into a real RFC. That is good enough to not impact this PR.

kislyuk · 2019-09-16T21:16:36Z

@diekhans I already imported it here: #113

rfcs/text/0000-bundle-definition.md

NoopDog · 2019-09-25T17:02:00Z

As a refinement and extension of this RFC I have created an alternate RFC linked here : RFC: HCA DCP Application Layer Bundle Types and Definitions .

The alternate proposal differs from this RFC mainly in that:

Type information is added to a new type.json metadata file rather than added to the DSS bundle.json file.
Type information is expressed in JSON rather than in RFC 7231 media type syntax.
The schema and bundle types are represented as JSON schema in the metadata schema repo and documented on the data portal rather than maintained in a DSS registry.
The proposed types are refined by making the process and protocol that created the bundle explicit in the type.json.

Your review/feedback is appreciated.

TimothyTickle

This looks much better, and the definitions feel less specific to use and a little more generalized and reusable (potentially less over-fit).

NoopDog · 2019-11-05T18:08:07Z

Pausing to merge with #119 and align with upcoming reproducibility and data citation requirements.

NoopDog · 2021-04-05T16:12:37Z

Obsolete

malloryfreeberg and others added 21 commits July 12, 2019 11:57

Added bundle definition RFC.

14c0b4b

Fixed formatting.

f58d817

Fixed formatting again.

f9daf4a

Added new bundle type description stubs.

6ce0cc0

Updated table header definitions.

c4f3735

Updated user story.

ed916a1

Fixed formatting.

e4c5e15

Added note about specification being OOS

e44d743

Updated motivation.

044a3bd

Updated use cases.

4f8e7bc

Added question about bundle definition.

5028e14

Removed bundle specifications.

055b9ff

Added notation of new bundle types.

67df958

Added clarification of new bundle types

f898c18

Updated use cases in table.

f05242e

Updated use cases in table.

8244ca9

Updated NB for new bundle types.

fa0a66e

Updated Resource bundle definition.

ff6921c

Update 0000-bundle-definition.md

efbcc3a

Added DAPS bundle type.

b6c8828

Updated questions.

bb551e7

malloryfreeberg added the Architecture label Jul 29, 2019

kislyuk requested changes Jul 30, 2019

View reviewed changes

kislyuk mentioned this pull request Jul 30, 2019

RFC: Bundle Types #86

Closed

Added reference to DAPS bundle RFC.

439cb91

malloryfreeberg mentioned this pull request Jul 31, 2019

RFC: Representing sequencing library preparations in the HCA DCP metadata standard #87

Merged

samanehsan reviewed Sep 6, 2019

View reviewed changes

kislyuk marked this pull request as ready for review September 11, 2019 16:50

TimothyTickle reviewed Sep 14, 2019

View reviewed changes

rfcs/text/0000-bundle-definition.md Outdated Show resolved Hide resolved

TimothyTickle reviewed Sep 14, 2019

View reviewed changes

TimothyTickle requested changes Sep 14, 2019

View reviewed changes

TimothyTickle reviewed Sep 14, 2019

View reviewed changes

kislyuk added 4 commits September 16, 2019 09:30

Merge #86 into #93

d968507

Fix typo, thanks @chmreid

7ab8aa6

Data Processing Service may consume secondary analysis bundles

8a448cb

Project bundles may be used by DCP services

f4d0722

kislyuk requested a review from lauraclarke September 16, 2019 16:48

xbrianh approved these changes Sep 16, 2019

View reviewed changes

rfcs/text/0000-bundle-definition.md Show resolved Hide resolved

rfcs/text/0000-bundle-definition.md Outdated Show resolved Hide resolved

NoopDog reviewed Sep 17, 2019

View reviewed changes

rfcs/text/0000-bundle-definition.md Show resolved Hide resolved

NoopDog reviewed Sep 17, 2019

View reviewed changes

rfcs/text/0000-bundle-definition.md Show resolved Hide resolved

brianraymor mentioned this pull request Sep 17, 2019

Processing Datasets that Span Multiple Data Collection Runs #88

Merged

NoopDog mentioned this pull request Sep 25, 2019

RFC: HCA DCP Domain Oriented Bundle Types and Definitions #119

Closed

TimothyTickle approved these changes Sep 26, 2019

View reviewed changes

kislyuk added the rfc-oversight-review label Sep 26, 2019

kislyuk added 2 commits September 27, 2019 07:37

Address review comments

a7b7953

Address review comments

b14ccb5

DailyDreaming approved these changes Sep 27, 2019

View reviewed changes

NoopDog added rfc-paused and removed rfc-oversight-review labels Nov 5, 2019

NoopDog closed this Apr 5, 2021

DailyDreaming deleted the mfreeberg-rfc-bundle-definitions branch April 15, 2021 17:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: HCA DCP Bundle Types and Definitions #93

RFC: HCA DCP Bundle Types and Definitions #93

malloryfreeberg commented Jul 29, 2019 •

edited by kislyuk

Loading

kislyuk Jul 30, 2019

malloryfreeberg Jul 31, 2019

diekhans Aug 5, 2019

TimothyTickle Sep 14, 2019

kislyuk Jul 30, 2019

malloryfreeberg Jul 31, 2019

samanehsan Sep 6, 2019

kislyuk Sep 11, 2019

NoopDog Sep 11, 2019

kislyuk Sep 12, 2019

NoopDog commented Sep 11, 2019

kislyuk commented Sep 11, 2019

TimothyTickle Sep 14, 2019

kislyuk Sep 16, 2019

TimothyTickle Sep 14, 2019

kislyuk Sep 16, 2019

TimothyTickle left a comment

TimothyTickle Sep 14, 2019

kislyuk commented Sep 16, 2019

diekhans commented Sep 16, 2019 •

edited

Loading

kislyuk commented Sep 16, 2019

NoopDog commented Sep 25, 2019

TimothyTickle left a comment

NoopDog commented Nov 5, 2019

NoopDog commented Apr 5, 2021


		Acceptance criteria are the conditions that a RFC must satisfy to be accepted by users or other stakeholders.

		1. Data consumers can make use of data from the DCP without knowing what a bundle is.


		> Asterisk (*) indicates bundles types that do not currently exist in the DCP but have been discussed as potential future bundle types to support data consumers.

		### New bundle types and their definitions

RFC: HCA DCP Bundle Types and Definitions #93

RFC: HCA DCP Bundle Types and Definitions #93

Conversation

malloryfreeberg commented Jul 29, 2019 • edited by kislyuk Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NoopDog commented Sep 11, 2019

kislyuk commented Sep 11, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TimothyTickle left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kislyuk commented Sep 16, 2019

diekhans commented Sep 16, 2019 • edited Loading

kislyuk commented Sep 16, 2019

NoopDog commented Sep 25, 2019

TimothyTickle left a comment

Choose a reason for hiding this comment

NoopDog commented Nov 5, 2019

NoopDog commented Apr 5, 2021

malloryfreeberg commented Jul 29, 2019 •

edited by kislyuk

Loading

diekhans commented Sep 16, 2019 •

edited

Loading