Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INFRA] Convert entity table to yaml #475

Merged
merged 53 commits into from
Aug 11, 2020
Merged

[INFRA] Convert entity table to yaml #475

Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
18a3f24
Draft entity and datatype files.
tsalo May 16, 2020
02c3403
Draft more specifications.
tsalo May 16, 2020
6ea0dfd
Add specifications for top-level files and directories.
tsalo May 16, 2020
6626368
Reorganize yaml files.
tsalo May 17, 2020
5c26565
Add inclusion for top level files.
tsalo May 19, 2020
d6601e7
Rename yaml files and add required field.
tsalo May 22, 2020
ee9535f
Convert dict to list in example file.
tsalo May 22, 2020
56a42b6
[DATALAD RUNCMD] Replace suffices with suffixes
yarikoptic May 22, 2020
596ed00
Add leading "." to extensions to match specification.
tsalo Jun 3, 2020
45fdfca
Convert dict to list.
tsalo Jun 3, 2020
9ec4e09
Fix iEEG extension based on issue.
tsalo Jun 3, 2020
d03d895
Split auxdatatypes and drop datatypes key.
tsalo Jun 3, 2020
8d3436c
Add descriptions to entities.
tsalo Jun 3, 2020
aa1ac3e
Draft metadata.
tsalo Jun 3, 2020
5791baf
ENH: top level bids.yaml to bind all separate files together
yarikoptic Jun 10, 2020
2193404
ENH: initial version of tools/bids-schema to assist with producing ma…
yarikoptic Jun 10, 2020
cc69735
RF: do not require !include - just load entire file tree!
yarikoptic Jun 10, 2020
c41a0dc
ENH: Produce entity table
yarikoptic Jun 10, 2020
797b23c
TEMP ENH: script to produce and replace the entity table inplace with…
yarikoptic Jun 11, 2020
56447c3
ENH: a few more paths for git to ignore
yarikoptic Jun 11, 2020
06429b5
[DATALAD RUNCMD] (Re)generate entity table from the schema
yarikoptic Jun 11, 2020
e58931d
Revert aa1ac3e.
tsalo Jun 13, 2020
ef49d03
Reorganize BIDS schema code.
tsalo Jun 14, 2020
69b50fe
Add modality names.
tsalo Jun 14, 2020
0673b73
Fix table header and allow run for anat.
tsalo Jun 14, 2020
b11685d
Add split entity.
tsalo Jun 14, 2020
8d0b612
Switch from click to argparse.
tsalo Jun 15, 2020
26f56ed
Merge branch 'master' into ref/json-entity
tsalo Jul 11, 2020
0bacc04
Split entity table and add docstrings.
tsalo Jul 18, 2020
e2b923b
Merge branch 'ref/json-entity' of https://github.com/tsalo/bids-speci…
tsalo Jul 18, 2020
532bb6c
Fix extra empty line.
tsalo Jul 18, 2020
4041d00
Apply suggestions from code review
tsalo Jul 22, 2020
bfac102
Remove top-level fieldmaps.
tsalo Jul 22, 2020
6ce57e1
Remove unsupported suffixes.
tsalo Jul 22, 2020
4e291e2
Fix 4d MEG suffix.
tsalo Jul 22, 2020
c9020f5
Merge branch 'master' into ref/json-entity
tsalo Jul 22, 2020
bf80162
Remove spurious trailing " in yaml paragraph
yarikoptic Jul 23, 2020
37132a4
Improve handling of multiline strings in yaml.
tsalo Jul 24, 2020
814e453
Add comments to separate groups.
tsalo Jul 24, 2020
0984f8c
Draft schema instructions in CONTRIBUTING.md.
tsalo Jul 24, 2020
cab708a
Partially address review.
tsalo Jul 26, 2020
8aa4994
Apply suggestions from code review
tsalo Jul 26, 2020
592a0db
Drop unnecessary paths from .gitignore.
tsalo Aug 2, 2020
1159d66
Try adding schema test.
tsalo Aug 2, 2020
5576c66
Use -c.
tsalo Aug 2, 2020
330e4dc
YAML isn't a builtin.
tsalo Aug 2, 2020
a233116
It's named pyyaml.
tsalo Aug 2, 2020
85ba0ca
Fix linting issues in yaml files and fix test.
tsalo Aug 2, 2020
dd4e1c1
Not a Python command.
tsalo Aug 2, 2020
125c492
Apply suggestions from code review
tsalo Aug 10, 2020
5c426b0
Implement changes from #556.
tsalo Aug 10, 2020
86650bf
Merge branch 'master' into ref/json-entity
tsalo Aug 10, 2020
177b15a
add LICENSE to top_level_files.yaml
sappelhoff Aug 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
site/
.DS_Store
src/.DS_Store
src/04-modality-specific-files/.DS_Store
src/04-modality-specific-files/.DS_Store
tsalo marked this conversation as resolved.
Show resolved Hide resolved
.idea
venvs
63 changes: 39 additions & 24 deletions src/99-appendices/04-entity-table.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,32 +1,47 @@
# Appendix IV: Entity table

This section compiles the entities (key-value pairs) described throughout this
specification, and establishes a common order within a filename. For example, if
a file has an acquisition and reconstruction label, the acquisition entity must
precede the reconstruction entity. REQUIRED and OPTIONAL entities for a given
file type are denoted. Entity formats indicate whether the value is alphanumeric
specification, and establishes a common order within a filename. 
For example, if a file has an acquisition and reconstruction label, the
acquisition entity must precede the reconstruction entity.
REQUIRED and OPTIONAL entities for a given file type are denoted. Entity
formats indicate whether the value is alphanumeric
tsalo marked this conversation as resolved.
Show resolved Hide resolved
(`<label>`) or numeric (`<index>`).

A general introduction to entities is given in the section on
[file name structure](../02-common-principles.md#file-name-structure)

| Entity | Subject | Session | Task | Acquisition | Contrast Enhancing Agent | Reconstruction | Phase-Encoding Direction | Run | Corresponding modality | Echo | Recording | Processed (on device) | Space | Split |
| :--------------------------------------------------------------------------------------------- | :------------ | :------------ | :------------- | :------------ | :----------------------- | :------------- | :----------------------- | :------------ | :--------------------- | :------------- | :------------------ | :-------------------- | :---------------| :-------------- |
| Format | `sub-<label>` | `ses-<label>` | `task-<label>` | `acq-<label>` | `ce-<label>` | `rec-<label>` | `dir-<label>` | `run-<index>` | `mod-<label>` | `echo-<index>` | `recording-<label>` | `proc-<label>` | `space-<label>` | `split-<index>` |
| anat<br>(T1w T2w T1rho T1map T2map T2star FLAIR FLASH PD PDmap PDT2 inplaneT1 inplaneT2 angio) | REQUIRED | OPTIONAL | | OPTIONAL | OPTIONAL | OPTIONAL | | | | | | | | |
| anat<br>(defacemask) | REQUIRED | OPTIONAL | | OPTIONAL | OPTIONAL | OPTIONAL | | | OPTIONAL | | | | | |
| func<br>(bold cbv phase sbref events) | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | OPTIONAL | OPTIONAL | OPTIONAL | OPTIONAL | | OPTIONAL | | | | |
| func<br>(physio stim) | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | | OPTIONAL | | OPTIONAL | | | OPTIONAL | OPTIONAL | | |
| dwi<br>(dwi bvec bval) | REQUIRED | OPTIONAL | | OPTIONAL | | | OPTIONAL | OPTIONAL | | | | | | |
| fmap<br>(phasediff phase1 phase2 magnitude1 magnitude2 magnitude fieldmap) | REQUIRED | OPTIONAL | | OPTIONAL | | | | OPTIONAL | | | | | | |
| fmap<br>(epi) | REQUIRED | OPTIONAL | | OPTIONAL | OPTIONAL | | REQUIRED | OPTIONAL | | | | | | |
| beh<br>(events stim physio) | REQUIRED | OPTIONAL | REQUIRED | | | | | | | | | | | |
| meg<br> | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | | | | OPTIONAL | | | | OPTIONAL | | OPTIONAL |
| eeg<br> | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | | | | OPTIONAL | | | | | | |
| ieeg<br> | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | | | | OPTIONAL | | | | | | |
| channels<br>(meg/eeg/ieeg) | REQUIRED | OPTIONAL | REQUIRED | | | | | OPTIONAL | | | | | | |
| headshape<br>(meg) | REQUIRED | OPTIONAL | | OPTIONAL | | | | | | | | | OPTIONAL | |
| markers<br>(meg) | REQUIRED | OPTIONAL | OPTIONAL | OPTIONAL | | | | | | | | | OPTIONAL | |
| photo<br>(meg/eeg/ieeg) | REQUIRED | OPTIONAL | | OPTIONAL | | | | | | | | | | |
| electrodes<br>(eeg/ieeg) | REQUIRED | OPTIONAL | | OPTIONAL | | | | | | | | | OPTIONAL | |
| events<br>(meg/eeg/ieeg) | REQUIRED | OPTIONAL | REQUIRED | | | | | OPTIONAL | | | | | | |
## Magnetic Resonance Imaging

| Entity | Subject | Session | Task | Acquisition | Contrast Enhancing Agent | Reconstruction | Phase-Encoding Direction | Run | Corresponding Modality | Echo | Recording | Processed (on device) |
|------------------------------------------------------------------------------------------------|---------------|---------------|----------------|---------------|----------------------------|------------------|----------------------------|---------------|--------------------------|----------------|---------------------|-------------------------|
| Format | `sub-<label>` | `ses-<label>` | `task-<label>` | `acq-<label>` | `ce-<label>` | `rec-<label>` | `dir-<label>` | `run-<index>` | `mod-<label>` | `echo-<index>` | `recording-<label>` | `proc-<label>` |
| anat<br>(T1w T2w T1rho T1map T2map T2star FLAIR FLASH PD PDmap PDT2 inplaneT1 inplaneT2 angio) | REQUIRED | OPTIONAL | | OPTIONAL | OPTIONAL | OPTIONAL | | OPTIONAL | | | | |
| anat<br>(defacemask) | REQUIRED | OPTIONAL | | OPTIONAL | OPTIONAL | OPTIONAL | | OPTIONAL | OPTIONAL | | | |
| dwi<br>(dwi sbref) | REQUIRED | OPTIONAL | | OPTIONAL | | | OPTIONAL | OPTIONAL | | | | |
| fmap<br>(phasediff phase1 phase2 magnitude1 magnitude2 magnitude fieldmap) | REQUIRED | OPTIONAL | | OPTIONAL | | | | OPTIONAL | | | | |
| fmap<br>(epi) | REQUIRED | OPTIONAL | | OPTIONAL | OPTIONAL | | REQUIRED | OPTIONAL | | | | |
| func<br>(bold cbv phase sbref events) | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | OPTIONAL | OPTIONAL | OPTIONAL | OPTIONAL | | OPTIONAL | | |
| func<br>(physio stim) | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | | OPTIONAL | | OPTIONAL | | | OPTIONAL | OPTIONAL |

## Encephalography (EEG, iEEG, and MEG)

| Entity | Subject | Session | Task | Acquisition | Run | Processed (on device) | Space | Split |
|----------------------------|---------------|---------------|----------------|---------------|---------------|-------------------------|-----------------|-----------------|
| Format | `sub-<label>` | `ses-<label>` | `task-<label>` | `acq-<label>` | `run-<index>` | `proc-<label>` | `space-<label>` | `split-<index>` |
| eeg<br>(eeg) | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | OPTIONAL | | | |
| ieeg<br>(ieeg) | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | OPTIONAL | | | |
| meg<br>(meg) | REQUIRED | OPTIONAL | REQUIRED | OPTIONAL | OPTIONAL | OPTIONAL | | OPTIONAL |
| meg<br>(headshape) | REQUIRED | OPTIONAL | | OPTIONAL | | | OPTIONAL | |
| meg<br>(markers) | REQUIRED | OPTIONAL | OPTIONAL | OPTIONAL | | | OPTIONAL | |
| channels<br>(meg eeg ieeg) | REQUIRED | OPTIONAL | REQUIRED | | OPTIONAL | | | |
| electrodes<br>(eeg ieeg) | REQUIRED | OPTIONAL | | OPTIONAL | | | OPTIONAL | |
| events<br>(meg eeg ieeg) | REQUIRED | OPTIONAL | REQUIRED | | OPTIONAL | | | |
| photo<br>(meg eeg ieeg) | REQUIRED | OPTIONAL | | OPTIONAL | | | | |

## Behavioral Data

| Entity | Subject | Session | Task |
|---------------------------------|---------------|---------------|----------------|
| Format | `sub-<label>` | `ses-<label>` | `task-<label>` |
| beh<br>(stim physio events beh) | REQUIRED | OPTIONAL | REQUIRED |
8 changes: 8 additions & 0 deletions src/schema/associated_data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- code/:
- required: False
- derivatives/:
- required: False
- sourcedata/:
- required: False
- stimuli/:
- required: False
14 changes: 14 additions & 0 deletions src/schema/auxdatatypes/channels.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
- datatypes:
- meg
- eeg
- ieeg
suffixes:
- channels
extensions:
- .json
- .tsv
entities:
sub: required
ses: optional
task: required
run: optional
13 changes: 13 additions & 0 deletions src/schema/auxdatatypes/electrodes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
- datatypes:
- eeg
- ieeg
suffixes:
- electrodes
extensions:
- .json
- .tsv
entities:
sub: required
ses: optional
acq: optional
space: optional
14 changes: 14 additions & 0 deletions src/schema/auxdatatypes/events.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
- datatypes:
- meg
- eeg
- ieeg
suffixes:
- events
extensions:
- .json
- .tsv
entities:
sub: required
ses: optional
task: required
run: optional
12 changes: 12 additions & 0 deletions src/schema/auxdatatypes/photo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
- datatypes:
- meg
- eeg
- ieeg
suffixes:
- photo
extensions:
- .jpg
entities:
sub: required
ses: optional
acq: optional
42 changes: 42 additions & 0 deletions src/schema/datatypes/anat.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# First group
- suffixes:
- T1w
- T2w
- T1rho
- T1map
- T2map
- T2star
- FLAIR
- FLASH
- PD
- PDmap
- PDT2
- inplaneT1
- inplaneT2
- angio
extensions:
- .nii.gz
- .nii
- .json
entities:
sub: required
ses: optional
run: optional
acq: optional
ce: optional
rec: optional
# Second group
- suffixes:
- defacemask
extensions:
- .nii.gz
- .nii
- .json
entities:
sub: required
ses: optional
run: optional
acq: optional
ce: optional
rec: optional
mod: optional
20 changes: 20 additions & 0 deletions src/schema/datatypes/beh.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
- suffixes:
- stim
- physio
extensions:
- .tsv.gz
- .json
entities:
sub: required
ses: optional
task: required
tsalo marked this conversation as resolved.
Show resolved Hide resolved
- suffixes:
- events
- beh
extensions:
- .tsv
- .json
entities:
sub: required
ses: optional
task: required
effigies marked this conversation as resolved.
Show resolved Hide resolved
26 changes: 26 additions & 0 deletions src/schema/datatypes/dwi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
- suffixes:
- dwi
extensions:
- .nii.gz
- .nii
- .json
- .bvec
- .bval
entities:
sub: required
ses: optional
acq: optional
dir: optional
run: optional
- suffixes:
- sbref
extensions:
- .nii.gz
- .nii
- .json
entities:
sub: required
ses: optional
acq: optional
dir: optional
run: optional
17 changes: 17 additions & 0 deletions src/schema/datatypes/eeg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
- suffixes:
- eeg
extensions:
- .json
- .edf
- .vhdr
- .vmrk
- .eeg
- .set
- .fdt
- .bdf
entities:
sub: required
ses: optional
task: required
acq: optional
run: optional
30 changes: 30 additions & 0 deletions src/schema/datatypes/fmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
- suffixes:
- phasediff
- phase1
- phase2
- magnitude1
- magnitude2
- magnitude
- fieldmap
extensions:
- .nii.gz
- .nii
- .json
entities:
sub: required
ses: optional
acq: optional
run: optional
- suffixes:
- epi
extensions:
- .nii.gz
- .nii
- .json
entities:
sub: required
ses: optional
acq: optional
ce: optional
dir: required
run: optional
49 changes: 49 additions & 0 deletions src/schema/datatypes/func.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
- suffixes:
- bold
- cbv
- phase
- sbref
extensions:
- .nii.gz
- .nii
- .json
entities:
sub: required
ses: optional
task: required
acq: optional
ce: optional
rec: optional
dir: optional
run: optional
echo: optional
- suffixes:
- events
extensions:
- .tsv
- .json
entities:
sub: required
ses: optional
task: required
acq: optional
ce: optional
rec: optional
dir: optional
run: optional
echo: optional
- suffixes:
- physio
- stim
extensions:
- .tsv.gz
- .json
entities:
sub: required
ses: optional
task: required
acq: optional
rec: optional
run: optional
recording: optional
proc: optional
tsalo marked this conversation as resolved.
Show resolved Hide resolved
18 changes: 18 additions & 0 deletions src/schema/datatypes/ieeg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
- suffixes:
- ieeg
extensions:
- .mefd/
- .json
- .edf
- .vhdr
- .eeg
- .vmrk
- .set
- .fdt
- .nwb
entities:
sub: required
ses: optional
task: required
acq: optional
run: optional
42 changes: 42 additions & 0 deletions src/schema/datatypes/meg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
- suffixes:
- meg
extensions:
- 4d/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<rant> wow, I didn't know that (sub)folders are allowed here. I truly think this was a step backward for meeg proposal to just allow all those formats -- there is no 'standardization' now, tools will just support some but not the other formats etc. </rant>

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume all extensions are supposed to have dots?

Suggested change
- 4d/
- .4d/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I look at it, it looks like the 4d format just has a folder with no suffix. Here's what it looks like:

sub-control01/
    ses-001/
        sub-control01_ses-001_scans.tsv
        meg/
            sub-control01_ses-001_coordsystem.json
            sub-control01_ses-001_headshape.pos
            sub-control01_ses-001_task-rest_run-01_meg/
                config
                hs_file
                e,rfhp1.0Hz.COH
                c,rfDC
            sub-control01_ses-001_task-rest_run-01_meg.json
            sub-control01_ses-001_task-rest_run-01_channels.tsv

I guess I should just put a / there, instead of 4d/ or .4d/?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this format is the biggest pain ever ... we should ask some MEG people what to do (in particular, Robert O. once he's back from vacation)

Can we have / with a comment that this is for BTi/4D data?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 4e291e2, but I'll hold off on resolving this in case further discussion is needed.

- .ds/
- .json
- .fif
- .sqd
- .con
- .raw
- .ave
- .mrk
- .kdf
- .mhd
entities:
sub: required
ses: optional
task: required
acq: optional
run: optional
proc: optional
split: optional
- suffixes:
- headshape
extensions:
- .pos
- .txt
entities:
sub: required
ses: optional
acq: optional
space: optional
- suffixes:
- markers
extensions:
- .json
entities:
sub: required
ses: optional
task: optional
acq: optional
space: optional
Loading