---

GDC Data Dictionary

The Genomic Data Commons’ (GDC) data dictionary provides the first level of validation for all data stored in and generated by the GDC. JSON schemas define all the individual entities (nodes) in the GDC data model. Moreover, these schemas define all of the relationships (links) between the nodes. Finally, the schemas define the valid key-value pairs that can be used to describe the nodes.

• GDC Data Dictionary
  • GDC Data Dictionary Structure
  • Node Properties and Examples
  • Dictionary Changes
    • Breaking Changes
    • Entity Relation Additions
    • Schema Additions
    • Cosmetic Corrections
    • Testing
    • Versioning
  • Gdcdatamodel2 auto generation
  • Setup pre-commit hook to check for secrets
  • Contributing

GDC Data Dictionary Structure

The GDC Data Model is covers all of the nodes within the GDC as well as the relationships between the different types of nodes. All of the nodes in the data model are strongly typed and individually defined for a specific data type. For example, submitted files can come in two different forms, aligned or unaligned; within the model we have two separately defined nodes for Submitted Unaligned Reads and Submitted Aligned Reads. Doing such allows for faster querying of the data model as well as providing a clear and concise representation of the data in the GDC.

Beyond node type, there are also a number of GDC extensions used to further define the nodes within the data model. Nodes are grouped up into categories that represent broad roles for the node such as analysis or biospecimen. Additionally, nodes are defined within their Program or Project and have descriptions of their use. All nodes also have a series of systemProperties; these properties are those that will be automatically filled by the system unless otherwise defined by the user. These basic properties define the node itself but it still needs to be placed into the model.

The model itself is represented as a graph. Within the schema are defined links; these links point from child to parent with Program being the root of the graph. The links also contain a backref that allows for a parent to point to a child. Other features of the link include a semantic label that describes the relationship between the two nodes, a multiplicity property that describes the numeric relationship from the child to the parent, and a requirement property to define whether a node must have that link. Taken all together the nodes and links create the directed graph of the GDC Data Model.

Node Properties and Examples

Each node contains a series of potential key-value pairs (properties) that can be used to characterize the data they represent. Some properties are categorized as required or preferred. If a submission lacks a required property, it cannot be accepted. Preferred properties can denote two things: the property is being highlighted as it has become more desired by the community or the property is being promoted to required. All properties not designated either required or preferred are still sought by GDC, but submissions without them are allowed.

The properties have further validation through their entries. Legal values are defined in each property. For the most part these are represented in the enum categories although some keys, such as submitter_id, will allow any string value as a valid entry. Other numeric properties can have maximum and minimum values to limit valid entries. For examples of what a valid entry would look like, each node has a mock submission located in the examples/valid/ directory.

Dictionary Changes

The following is an attempt to layout guidelines for the level of impact of changes to the dictionary by categorizing them into Breaking Changes, Entity Relation Additions, Schema Additions, Cosmetic Corrections.

Breaking Changes

Breaking changes are changes to the dictionary such that previously allowable data is invalid against the new schema, e.g. a removal of part of the dictionary.

N.B. That not all changes classified here as Breaking Changes are promised to require a data migration. It is possible that no data exists in the GDC that is invalidated by the change, e.g. making a field required that has never been left blank. This should be confirmed against the corpus of data and the userbase should be notified of a break in backwards-compatibility.

Breaking Changes include:

• Renaming/removing anything that is not a description or comment
  • Removing an entity schema
  • Removing a property's allowed type
  • Removing a property's allowed enum value
  • Changing an entity's category
  • Changing an entity's unique_keys
  • Changing an entity's links, including label, backref
  • Removing a property from an entity schema
• Changing existence requirements
  • Adding a property to the required list
  • Changing link required from false to true
  • Changing link multiplicity from one_to_many or many_to_one to one_to_one
  • Changing link subgroup exclusivity from false to true

Handling breaking changes:

Sometimes it may be best to introduce necessary breaking changes incrementally. Given you have State A and State B, which are incomatible, if you can create a State AB that is compatible with both, you can upgrade to State AB without breaking changes, update data to be compliant with State B, then upgrade to State B.

1. State A is deployed
2. Upgrade to State AB
3. Update data while State AB is deployed to be valid under State B
4. Upgrade to State B

An example could be: Introduce required property color:

1. Property color does not exist
2. Deploy schema that allows but does not require color
3. Add color to all records
4. Deploy schema that requires color

Entity Relation Additions

Additions to the dictionary that create entities or add links between entities should not be considered breaking changes, however, they should be carefully considered in context of downstream effects.

Entity Relation Additions include:

• Adding a new entity schema
• Adding a new link between entities

Entity Relation downstream effects:

• The GDC will have to update the database schema
• Users should be notified of additions

Schema Additions

The GDC is setup to allow strict additions to properties have minimal impact on existing data.

Schema Additions include:

• New properties
• New allowed types for properties
• New allowed enum members for properties

Schema Addition downstream effects:

• Users should be notified of additions

Cosmetic Corrections

Cosmetic corrections are changes that have little to no behavioral effects.

Cosmetic Corrections include:

• Changes to terms
• Changes to documentation
• Schema formatting changes

Schema Addition downstream effects:

• No large impacts

Testing

Commits will automagically be run on TravisCI when a Pull Request is opened. If you would like to test locally they are run via tox

Versioning

The GDC Dictionary should follow Semantic Versioning by updating the line in setup.py file to MAJOR.MINOR.PATCH accordingly:

1. MAJOR: version when you make incompatible API changes: Breaking Changes
   • e.g. 1.2.4 -> 2.0.0
2. MINOR: version when you add functionality in a backwards-compatible manner: Relationship Additions, Schema Additions
   • e.g. 1.2.4 -> 1.3.0
3. PATCH: version when you make backwards-compatible bug fixes: Cosmetic Corrections
   • e.g. 1.2.4 -> 1.2.5

Gdcdatamodel2 auto generation

Gdcdatamodel2 should be generated automatically on gitlab for each commit push. The generated python artifact should be in https://nexus.osdc.io/#browse/browse:pypi-snapshots:gdcdatamodel2

The gitlab pipline also automatically push a new branch to gdcdatamodel2 on github.

If you want, you can also manually run the pipeline to generate a new version.

1. Go to https://gitlab.datacommons.io/nci-gdc/development/gdcdictionary/-/pipelines/new
2. Select the branch/tag of gdcdictionary you want to use, default: develop
3. (optional) the generated version of gdcdatamodel2 should based on the branch/tag you selected in previous step. But if you want to generate from a different branch, change GDCDICTIONARY_TARGET_VERSION_OVERRIDE in the variables.
4. click Run Pipeline button.

Setup pre-commit hook to check for secrets

We use pre-commit to setup pre-commit hooks for this repo. We use detect-secrets to search for secrets being committed into the repo.

To install the pre-commit hook, run

pre-commit install

To update the .secrets.baseline file run

detect-secrets scan --update .secrets.baseline

.secrets.baseline contains all the string that were caught by detect-secrets but are not stored in plain text. Audit the baseline to view the secrets .

detect-secrets audit .secrets.baseline

Contributing

Read how to contribute here.