Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INFRA] Convert entity table to yaml #475

Merged
merged 53 commits into from
Aug 11, 2020
Merged

[INFRA] Convert entity table to yaml #475

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
18a3f24
Draft entity and datatype files.
tsalo May 16, 2020
02c3403
Draft more specifications.
tsalo May 16, 2020
6ea0dfd
Add specifications for top-level files and directories.
tsalo May 16, 2020
6626368
Reorganize yaml files.
tsalo May 17, 2020
5c26565
Add inclusion for top level files.
tsalo May 19, 2020
d6601e7
Rename yaml files and add required field.
tsalo May 22, 2020
ee9535f
Convert dict to list in example file.
tsalo May 22, 2020
56a42b6
[DATALAD RUNCMD] Replace suffices with suffixes
yarikoptic May 22, 2020
596ed00
Add leading "." to extensions to match specification.
tsalo Jun 3, 2020
45fdfca
Convert dict to list.
tsalo Jun 3, 2020
9ec4e09
Fix iEEG extension based on issue.
tsalo Jun 3, 2020
d03d895
Split auxdatatypes and drop datatypes key.
tsalo Jun 3, 2020
8d3436c
Add descriptions to entities.
tsalo Jun 3, 2020
aa1ac3e
Draft metadata.
tsalo Jun 3, 2020
5791baf
ENH: top level bids.yaml to bind all separate files together
yarikoptic Jun 10, 2020
2193404
ENH: initial version of tools/bids-schema to assist with producing ma…
yarikoptic Jun 10, 2020
cc69735
RF: do not require !include - just load entire file tree!
yarikoptic Jun 10, 2020
c41a0dc
ENH: Produce entity table
yarikoptic Jun 10, 2020
797b23c
TEMP ENH: script to produce and replace the entity table inplace with…
yarikoptic Jun 11, 2020
56447c3
ENH: a few more paths for git to ignore
yarikoptic Jun 11, 2020
06429b5
[DATALAD RUNCMD] (Re)generate entity table from the schema
yarikoptic Jun 11, 2020
e58931d
Revert aa1ac3e.
tsalo Jun 13, 2020
ef49d03
Reorganize BIDS schema code.
tsalo Jun 14, 2020
69b50fe
Add modality names.
tsalo Jun 14, 2020
0673b73
Fix table header and allow run for anat.
tsalo Jun 14, 2020
b11685d
Add split entity.
tsalo Jun 14, 2020
8d0b612
Switch from click to argparse.
tsalo Jun 15, 2020
26f56ed
Merge branch 'master' into ref/json-entity
tsalo Jul 11, 2020
0bacc04
Split entity table and add docstrings.
tsalo Jul 18, 2020
e2b923b
Merge branch 'ref/json-entity' of https://github.com/tsalo/bids-speci…
tsalo Jul 18, 2020
532bb6c
Fix extra empty line.
tsalo Jul 18, 2020
4041d00
Apply suggestions from code review
tsalo Jul 22, 2020
bfac102
Remove top-level fieldmaps.
tsalo Jul 22, 2020
6ce57e1
Remove unsupported suffixes.
tsalo Jul 22, 2020
4e291e2
Fix 4d MEG suffix.
tsalo Jul 22, 2020
c9020f5
Merge branch 'master' into ref/json-entity
tsalo Jul 22, 2020
bf80162
Remove spurious trailing " in yaml paragraph
yarikoptic Jul 23, 2020
37132a4
Improve handling of multiline strings in yaml.
tsalo Jul 24, 2020
814e453
Add comments to separate groups.
tsalo Jul 24, 2020
0984f8c
Draft schema instructions in CONTRIBUTING.md.
tsalo Jul 24, 2020
cab708a
Partially address review.
tsalo Jul 26, 2020
8aa4994
Apply suggestions from code review
tsalo Jul 26, 2020
592a0db
Drop unnecessary paths from .gitignore.
tsalo Aug 2, 2020
1159d66
Try adding schema test.
tsalo Aug 2, 2020
5576c66
Use -c.
tsalo Aug 2, 2020
330e4dc
YAML isn't a builtin.
tsalo Aug 2, 2020
a233116
It's named pyyaml.
tsalo Aug 2, 2020
85ba0ca
Fix linting issues in yaml files and fix test.
tsalo Aug 2, 2020
dd4e1c1
Not a Python command.
tsalo Aug 2, 2020
125c492
Apply suggestions from code review
tsalo Aug 10, 2020
5c426b0
Implement changes from #556.
tsalo Aug 10, 2020
86650bf
Merge branch 'master' into ref/json-entity
tsalo Aug 10, 2020
177b15a
add LICENSE to top_level_files.yaml
sappelhoff Aug 11, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions src/schema/associated_data.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
- code/:
- required: False
- derivatives/:
- required: False
- sourcedata/:
- required: False
- stimuli/:
- required: False
45 changes: 45 additions & 0 deletions src/schema/datatypes/anat.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
groups:
tsalo marked this conversation as resolved.
Show resolved Hide resolved
# First group
- datatypes:
- anat
suffixes:
- T1w
- T2w
- T1rho
- T1map
- T2map
- T2star
- FLAIR
- FLASH
- PD
- PDmap
- PDT2
- inplaneT1
- inplaneT2
- angio
extensions:
- nii.gz
tsalo marked this conversation as resolved.
Show resolved Hide resolved
- nii
- json
entities:
sub: required
ses: optional
acq: optional
ce: optional
rec: optional
# Second group
- datatypes:
- anat
suffixes:
- defacemask
extensions:
- nii.gz
- nii
- json
entities:
sub: required
ses: optional
acq: optional
ce: optional
rec: optional
mod: optional
26 changes: 26 additions & 0 deletions src/schema/datatypes/beh.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
group1:
datatypes:
- beh
suffixes:
- stim
- physio
extensions:
- tsv.gz
- json
entities:
sub: required
ses: optional
task: required
tsalo marked this conversation as resolved.
Show resolved Hide resolved
group2:
datatypes:
- beh
suffixes:
- events
- beh
extensions:
- tsv
- json
entities:
sub: required
ses: optional
task: required
effigies marked this conversation as resolved.
Show resolved Hide resolved
15 changes: 15 additions & 0 deletions src/schema/datatypes/channels.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
group1:
datatypes:
- meg
- eeg
- ieeg
suffixes:
- channels
extensions:
- json
- tsv
entities:
sub: required
ses: optional
task: required
run: optional
32 changes: 32 additions & 0 deletions src/schema/datatypes/dwi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
group1:
datatypes:
- dwi
suffixes:
- dwi
extensions:
- nii.gz
- nii
- json
- bvec
- bval
entities:
sub: required
ses: optional
acq: optional
dir: optional
run: optional
group2:
datatypes:
- dwi
suffixes:
- sbref
extensions:
- nii.gz
- nii
- json
entities:
sub: required
ses: optional
acq: optional
dir: optional
run: optional
20 changes: 20 additions & 0 deletions src/schema/datatypes/eeg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
group1:
datatypes:
- eeg
suffixes:
- eeg
extensions:
- json
- edf
- vhdr
- vmrk
- eeg
- set
- fdt
- bdf
entities:
sub: required
ses: optional
task: required
acq: optional
run: optional
14 changes: 14 additions & 0 deletions src/schema/datatypes/electrodes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
group1:
datatypes:
- eeg
- ieeg
suffixes:
- electrodes
extensions:
- json
- tsv
entities:
sub: required
ses: optional
acq: optional
space: optional
15 changes: 15 additions & 0 deletions src/schema/datatypes/events.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
group1:
datatypes:
- meg
- eeg
- ieeg
suffixes:
- events
extensions:
- json
- tsv
entities:
sub: required
ses: optional
task: required
run: optional
36 changes: 36 additions & 0 deletions src/schema/datatypes/fmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
group1:
datatypes:
- fmap
suffixes:
- phasediff
- phase1
- phase2
- magnitude1
- magnitude2
- magnitude
- fieldmap
extensions:
- nii.gz
- nii
- json
entities:
sub: required
ses: optional
acq: optional
run: optional
group2:
datatypes:
- fmap
suffixes:
- epi
extensions:
- nii.gz
- nii
- json
entities:
sub: required
ses: optional
acq: optional
ce: optional
dir: required
run: optional
58 changes: 58 additions & 0 deletions src/schema/datatypes/func.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
group1:
datatypes:
- func
suffixes:
- bold
- cbv
- phase
- sbref
extensions:
- nii.gz
- nii
- json
entities:
sub: required
ses: optional
task: required
acq: optional
ce: optional
rec: optional
dir: optional
run: optional
echo: optional
group2:
datatypes:
- func
suffixes:
- events
extensions:
- tsv
- json
entities:
sub: required
ses: optional
task: required
acq: optional
ce: optional
rec: optional
dir: optional
run: optional
echo: optional
group3:
datatypes:
- func
suffixes:
- physio
- stim
extensions:
- tsv.gz
- json
entities:
sub: required
ses: optional
task: required
acq: optional
rec: optional
run: optional
recording: optional
proc: optional
tsalo marked this conversation as resolved.
Show resolved Hide resolved
21 changes: 21 additions & 0 deletions src/schema/datatypes/ieeg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
group1:
datatypes:
- ieeg
suffixes:
- ieeg
extensions:
- json
- edf
- vhdr
- eeg
- vmrk
- set
- fdt
- nwb
- mef
entities:
sub: required
ses: optional
task: required
acq: optional
run: optional
50 changes: 50 additions & 0 deletions src/schema/datatypes/meg.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
group1:
datatypes:
- meg
suffixes:
- meg
extensions:
- json
- ctf/
- fif
- 4d/
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

<rant> wow, I didn't know that (sub)folders are allowed here. I truly think this was a step backward for meeg proposal to just allow all those formats -- there is no 'standardization' now, tools will just support some but not the other formats etc. </rant>

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume all extensions are supposed to have dots?

Suggested change
- 4d/
- .4d/

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that I look at it, it looks like the 4d format just has a folder with no suffix. Here's what it looks like:

sub-control01/
    ses-001/
        sub-control01_ses-001_scans.tsv
        meg/
            sub-control01_ses-001_coordsystem.json
            sub-control01_ses-001_headshape.pos
            sub-control01_ses-001_task-rest_run-01_meg/
                config
                hs_file
                e,rfhp1.0Hz.COH
                c,rfDC
            sub-control01_ses-001_task-rest_run-01_meg.json
            sub-control01_ses-001_task-rest_run-01_channels.tsv

I guess I should just put a / there, instead of 4d/ or .4d/?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, this format is the biggest pain ever ... we should ask some MEG people what to do (in particular, Robert O. once he's back from vacation)

Can we have / with a comment that this is for BTi/4D data?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 4e291e2, but I'll hold off on resolving this in case further discussion is needed.

- sqd
- con
- raw
- ave
- mrk
- kdf
- mhd
entities:
sub: required
ses: optional
task: required
acq: optional
run: optional
proc: optional
group2:
datatypes:
- meg
suffixes:
- headshape
extensions:
- pos
- txt
entities:
sub: required
ses: optional
acq: optional
space: optional
group3:
datatypes:
- meg
suffixes:
- markers
extensions:
- json
entities:
sub: required
ses: optional
task: optional
acq: optional
space: optional
13 changes: 13 additions & 0 deletions src/schema/datatypes/photo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
group1:
datatypes:
- meg
- eeg
- ieeg
suffixes:
- photo
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dear @tsalo -- Thank you for leading this effort! It is really fun to look at BIDS in such as structured way!

So within this datatypes/ we indeed have datatypes which gained their folder/ within BIDS hierarchy (they also have themselves duplicated now in datatypes) such as anat/, func/, fmap/, etc. (+ beh/ which isn't imaging), and then we have another group -- those which could have indeed been a modality of their own, but they are not "imaging data". They became auxiliary files which could go along with base imaging (and beh/) datatypes. I think that

  • all the photo, channels etc should get their own "folder" in this schema, e.g. auxiliary or auxdatatypes, and be moved there (instead of being under datatypes)
  • datatypes/ entries can get rid of the datatypes field in their .yaml files -- they should then have 1-to-1 correspondence from the filename to corresponding datatype.

On the top level (for inclusion) I see smth like

datatypes:
  anat: !include datatypes/anat.yaml
  func: !include datatypes/func.yaml
  ...
auxdatatypes:
  channels: !include auxdatatypes/channels.yaml
  event: !include auxdatatypes/events.yaml

This would make it easier to tell one (which gets its own folder in BIDS) from another (just provides additional suffixed files).

Alternative could be - keep as is, and then add additional field (in a yet to be created higher level "inclusion" file) to signal which datatype is auxiliary, but then it would be a bit less structured IMHO and most likely we would need to maintain that redundant self mentioning within datatypes for the base data types.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I split auxdatatypes from datatypes and dropped the datatypes key from the datatype yamls. I have to say, it looks much cleaner.

extensions:
- jpg
entities:
sub: required
ses: optional
acq: optional
Loading