BIDSify magnitude data and labels

[jcohenadad]

Motivation: we want to upload the data to OpenNeuro, so we need to BIDSify them.

The magnitude data are already in BIDS and are located here (internal server): https://data.neuro.polymtl.ca/datasets/whole-spine

The segmentation labels organized by @CharlesPageot are located here.

Here's a tree:

├── whole-spine_air_tissue_labels  --> body, lungs, trachea, sinuses
├── whole-spine_bone_discs_labels --> vertebrae, intervertebral discs
├── whole-spine_samseg_labels --> a bunch of labels in the brain, from SamSeg output. 
└── whole-spine_spine_labels --> spinal canal or spinal cord --> @CharlesPageot ?

They need to be organized according to BIDS, but for that we need to agree on the organization. Few thoughts:

Additional notes:

• the README.md seems inaccurate at some places.
• whole-spine_samseg_labels folder name is misleading
• We will need to merge the magnitude data (under 'root' of BIDS folder) and the segmentations (under 'derivatives/labels')
• We will need to decide how we want to combine all the segmentation files. This will depend on how @CharlesPageot 's code to process these data look like. @CharlesPageot can you please upload your code on GH asap so we can start organizing the files accordingly.

Important

The magnitude BIDS data already include some labels, that were likely generated by @NathanMolinier. I will do some digging to understand what labels already exist, and which ones are duplicates of what @CharlesPageot shared in the GDrive folder.

[jcohenadad]

OK, good news, the dataset and some of the labels are already in BIDS format. More details:

• sub-amuAL_T1w_label-brain_dseg.nii.gz --> SAMSEG labels head labels
• sub-amuAL_T1w_label-CSF_seg --> CSF
• sub-amuAL_T1w_label-SC_seg --> spinal cord
• sub-amuAL_T1w_label-spine_dseg --> discrete vertebrae + intervertebral discs (one value per level)

So, what we need to do next, is to develop code that:

• cleans up the SAMSEG labels, ie: only keep the important ones (eye, skull, fat?) @CharlesPageot your input needed here. You mentioned you did this in Slicer3D, what exactly did you do, and where did you put the corrected labels? Are the corrected labels under the GDrive folder whole-spine_samseg_labels or does this folder contain the direct (ie not manually corrected) output from SAMSEG?
• (MB) takes the labels from whole-spine_air_tissue_labels (from gdrive, lungs, trachea, sinuses) and add them to the git-annex dataset
  • https://data.neuro.polymtl.ca/datasets/whole-spine/pulls/5
• (MB) assembles all the labels into a single NIfTI file, with tissue value that are consistent with https://github.com/shimming-toolbox/tissue-to-MRproperty?tab=readme-ov-file#look-up-table
  • branch: mb/single-nifti
• add those labels to the git-annex dataset
• package and upload to OpenNeuro

[mathieuboudreau]

assembles all the labels into a single NIfTI file, with tissue value that are consistent with https://github.com/shimming-toolbox/tissue-to-MRproperty?tab=readme-ov-file#look-up-table

From a quick check, it looks like some of the air-tissue labels will overlap with the SAMSEG generated labels, eg,

Though maybe this won't be an issue after first point is done (cleans up the SAMSEG labels, ie: only keep the important ones (eye, skull, fat?)), we should make sure to write code that checks this and decide how to handle it (unless we want a 4D nifti, but my guess is no)

[mathieuboudreau]

The T1w/T2w images of the subjects have not been defaced,

https://docs.openneuro.org/user_guide.html
Uploading your dataset
4. You will need to ensure that either all structural scans have been defaced (we recommend the pydeface tool) or that you have explicit participant consent and ethical authorization to publish without defacing.

Considering that information, should deface we the T1w/T2w volumes prior to uploading to OpenNeuro?

[mathieuboudreau]

takes the labels from whole-spine_air_tissue_labels (from gdrive, lungs, trachea, sinuses) and add them to the git-annex dataset

Working on this

[mathieuboudreau]

The segmentation labels organized by @CharlesPageot are located here.

sub-unfErssm021 is missing their air-tissue labels in the GDrive, @CharlesPageot are you aware and if so, why?

[mathieuboudreau]

The segmentation labels organized by @CharlesPageot are located here.

sub-unfErssm021 is missing their air-tissue labels in the GDrive, @CharlesPageot are you aware and if so, why?

Ah - that subject only had T2w data.

There's other subjects missing labels due to the raw data, eg sub-unfErssm001 has no brain labels due to the mispositioned T1w volume. Something to consider while coding the merging/splitting of labels

[mathieuboudreau]

takes the labels from whole-spine_air_tissue_labels (from gdrive, lungs, trachea, sinuses) and add them to the git-annex dataset

Working on this

Labels added to branch mb/air-tissue-labels, PR opened (https://data.neuro.polymtl.ca/datasets/whole-spine/pulls/5), but its a WIP because I'm missing json sidecar information:

• date labels were computed
• totalseg version that was used

Who would know this, @CharlesPageot ?

[sriosq]

Hello @mathieuboudreau. Regarding the conflicting labels. With the code in tissue to MR property, we have implemented a function that checks if all the pixels have values according to the dictionary of labels used.

[mathieuboudreau]

assembles all the labels into a single NIfTI file, with tissue value that are consistent with https://github.com/shimming-toolbox/tissue-to-MRproperty?tab=readme-ov-file#look-up-table

On this now.

assembles all the labels into a single NIfTI file, with tissue value that are consistent with https://github.com/shimming-toolbox/tissue-to-MRproperty?tab=readme-ov-file#look-up-table

This instruction wasn't clear to me - at first you say to merge all the labels into one file of labels, but after say that the tissue values (in the nifti file?) should be consistent with that table.

Do you mean,

1. simply assemble all the labels into a single nifti files, such that the label values in that file are consistent with that repo's label value definition for all tissues, i.e. these dictionaries?

or

2. assemble all the labels into a single nifti file, and then using that label file generate additional nifti files for each MR property (i.e. for each subject, generate whole-body label file +
   T1, T2, T2*, PD, and Susceptibility files)?

Regardless, I can start working on it and adjust as needed

[jcohenadad]

Considering that information, should deface we the T1w/T2w volumes prior to uploading to OpenNeuro?

No, this will defeat the purpose of the project (ie: design shim coils for the head-- if part of the head is missing, then that's problematic. Let's hear back from the Marseille folks what they say.

[jcohenadad]

date labels were computed
totalseg version that was used
Who would know this, @CharlesPageot ?

Possibly @Nilser3 and/or @NathanMolinier

[jcohenadad]

simply assemble all the labels into a single nifti files, such that the label values in that file are consistent with that repo's label value definition for all tissues, i.e. these dictionaries?

yes, that's exactly what I meant, sorry for the lack of clarity

[jcohenadad]

I just noticed that whole-spine_air_tissue_labels also has the body as a class (I've edited #4 (comment) to add the info).

So I think that the only labels to extract from the SAMSEG segmentation are the skull and the eyes. More details here #5

[NathanMolinier]

I am a bit late on the issue but here are some comments:

• Currently our derivatives convention is not completely BIDS (e.g. _T1w should be replaced by _mod-T1w and labels should be contained in separate folders) but right now BIDS validator is only looking at the root folder so maybe it will be fine for open neuro.
• If we decide to go with discrete segmentation files (_dseg) then for clarity we should stick with BIDS convention and add a file containing the mapping (see Common image-derived labels). If we don't want to add this file everywhere, we can also add the file at the root of the derivatives/labels and point to it when necessary using a custom entity (e.g. _seg-airTissue) in the filename (e.g. _seg-Desikan here Discrete Segmentations)
• Also about the name whole-spine, since we do not cover all the spine maybe we could rename it whole-spinalcord ?

[mathieuboudreau]

Just writing some additional notes for myself now:

1. Some labels values defined in the label represent more than one tissue, eg.in the example below label = 2 means both white matter (in *label-brain_dseg.nii.gz) and the sinuses (in *label-air_tissue.nii.gz)

Green = 2, blue=everything else

2. A lot of label values in the subject label files are undefined in the dictionnaries of the tissue-to-MRproperty repo, eg:

• All the different brain regions/tissues (there's only "brain" in the dictionnaires, but not left/right WM, gm, etc)
• All the different SC labels

So my assumption is that I'll need to write up my own dictionary that maps the specific tissues labels from the subject files to the more general labels in the dictionaries (eg, left white matter, right white matter, gm, etc would all be converted to the label for just "brain").

That dictionary I'll create will also be where we can mask out some of the SAMSEG labels per Julien's first point #4 (comment)

But also, this brings up two more issues:

1. the dictionaries is not setup for a "whole-body" label dataset (yet?), as it asks for the different "tool" used, and...

2. the same tissue has multiple label values in some instances (i.e. the trachea is defined as a label 16 for TotalSeg_CT and label value 4 for "charles"), but also...

3. the same label value can mean multiple tissues according to that file (i.e. label value 2 is the left kidney for "TotalSeg_MRI" but the sinus for "charles").

So I think there will need a new "full body" (for us) definition of the label dictionnaries (eg. an overriding BIDS label tsv files as linked by Nathan, https://bids-specification.readthedocs.io/en/stable/derivatives/imaging.html#common-image-derived-labels) and then map our final merged labels to all their chi values as defined in tissue-to-MRprop, or define a dictionnary that maps out (new) labels to the tool + label combo already existing in https://github.com/shimming-toolbox/tissue-to-MRproperty/blob/main/functions/utils/select_tool.py so that they can be looked up as needed.

[CharlesPageot]

To clarify @jcohenadad, the whole-spine_samseg_labels contains the direct output of SAMSEG (no correction). I made a python script for Slicer that keeps only the relevant labels (skull and eyes) and perform a smoothing on the skull. Here is a preview of what the smoothing does:

I uploaded the script in this branch and the labels with only the skull and eyes are added here in the GDrive.

[jcohenadad]

Great! Thank you @CharlesPageot, this is super helpful. So, I think we can start from there, and I suggest maybe increasing the smoothing kernel. @mathieuboudreau do you want to look into this? thanks!!

[jcohenadad]

A lot of label values in the subject label files are undefined in the dictionnaries of the tissue-to-MRproperty repo, eg:

the purpose of getting these segmentations into MRproperties is to associate each segmentation with a susceptibility value. So it doesn't matter if the term "body" does not exist in the MRproperty dictionary. What we need to do is to associate the "closest" tissue in terms of susceptibility. Eg: we could consider that "body" is "muscle" (or any other tissue that makes sense). @CharlesPageot what did you choose?

same comment for the eyes (i guess this is like water? or maybe there is a specific susceptibility value for it?)

[CharlesPageot]

When I created the susceptibility maps, I asked @sriosq to make a specific dictionary for my project. It can be seen inside this .py file and it's the tool called 'charles'. Here we should definitely find another name for the dicto to reference this project.

[mathieuboudreau]

I uploaded the script in this branch and the labels with only the skull and eyes are added here in the GDrive.

Just writing a note: the filenames need to be fixed for BIDS comparitbility

Reason: extra _ between T1w and label, i.e.

sub-amuAL_T1w__label-skull-eyes_dseg.nii.gz --> sub-amuAL_T1w_label-skull-eyes_dseg.nii.gz

[NathanMolinier]

Labels added to branch mb/air-tissue-labels, PR opened (https://data.neuro.polymtl.ca/datasets/whole-spine/pulls/5), but its a WIP because I'm missing json sidecar information:

• date labels were computed
• totalseg version that was used

Who would know this, @CharlesPageot ?

This was a custom nnUNet model. More information about the training are available here: duke:/projects/fieldmap-totalsegmentator

[NathanMolinier]

Just writing a note: the filenames need to be fixed for BIDS comparitbility

Reason: extra _ between T1w and label, i.e.

sub-amuAL_T1w__label-skull-eyes_dseg.nii.gz --> sub-amuAL_T1w_label-skull-eyes_dseg.nii.gz

There are actually 4 problems here from a BIDS perspective:

1. _T1w should be replaced with _mod-T1w (not that problematic)
2. Double underscore is indeed wrong (problematic)
3. _label-skull-eyes should be replaced by something like _label-skullEyes (Attach only one value to a key: _key-value_key-value_key-value etc.) (problematic)
4. Technically the key _label- should be replaced with _seg- (not that problematic)

[mathieuboudreau]

Just writing a note: the filenames need to be fixed for BIDS comparitbility
Reason: extra _ between T1w and label, i.e.
sub-amuAL_T1w__label-skull-eyes_dseg.nii.gz --> sub-amuAL_T1w_label-skull-eyes_dseg.nii.gz

There are actually 4 problems here from a BIDS perspective:

1. _T1w should be replaced with _mod-T1w (not that problematic)
2. Double underscore is indeed wrong (problematic)
3. _label-skull-eyes should be replaced by something like _label-skullEyes (Attach only one value to a key: _key-value_key-value_key-value etc.) (problematic)
4. Technically the key _label- should be replaced with _seg- (not that problematic)

Thanks! A lot of those would then apply to the other label files, this'll need to be handled in https://data.neuro.polymtl.ca/datasets/whole-spine/pulls/5

[NathanMolinier]

I will go through the PR to double check for errors before merging

[mathieuboudreau]

Great! Thank you @CharlesPageot, this is super helpful. So, I think we can start from there, and I suggest maybe increasing the smoothing kernel. @mathieuboudreau do you want to look into this? thanks!!

@jcohenadad Here's the best I could accomplish for sub-amuAL-

orientation 1

orientation 2

orientation 3

orientation 4

To achieve this, here are the steps I followed in Slicer's segmentation editor:

1- Keep largest islands (min size = 1000 vox)
2- Smoothing - Closing (fill holes) -> 15 mm
3- Margin - Grow (2 mm)
4- Smoothing - Closing (fill holes) -> 15 mm
5- Margin - Shrink (1 mm)

I tried with just increasing smoothing to larger and larger numbers, and tho it eventually closed all the skull's holes ~35mm-45mm smoothing factor, it did some strange thing around the ears wear the "skull" would pertrude, see:

I played around with FSL also, but couldn't achieve better result than with the slicer tools.

Though these settings filled-in this subject fairly well I think, I'm unsure of how it will perform through all subjects. But the key for me to finally get better results was to use the "margin-grow" to fill in the holes a bit more so that the 2nd smoothing-closing could finally fill in the remaining holes nicely. Margin-shrink was necessary to brign back down the thickness of everything, but found that if I set the shrink to the same value as the grow (2mm), some small holes would reappear.

Let me know what you think; I did this subject manually, but I think scripting it like Charles did should be straightforward and I could try running it on all subjects to see if the parameters are about right for most of them

p.s. I've not yet explore improving the eyes.

[jcohenadad]

This is great @mathieuboudreau ! Let's see how it performs on the other subjects.

[mathieuboudreau]

Here's a gif showing the smoothened set of skulls for all subjects using the script I wrote,

Some notes:

• Some small holes remain in some skulls, would need manual finetuning to fix
• If the skull was too close to an edge of the FOV, the "fill holes" algorithm would occasionally stretch it to the edge of the FOV
• The skull is a bit thicker (since I do +2mm margin but then only -1 margin), however shrinking it any more reintroduces holes in the skull, so this is the tradeoff
• Occasionally there's clear overlap of the brain, but not often
• The sinuses fill out as if they're bone during the fill hole process, but those will be reintroduced using the sinus mask
• Skull regions where the SAMSEG segmentation had very large holes typically were thinner after the smoothing out of everything because of how the fill holes smoothing works; I don't think there's a way to auto-thicken regions, only the whole segmentation
• A couple skull still are completely bad, however their SAMSEG segmentation were quite bad in the first place; garbage in, garbage out,, eg,

[jcohenadad]

This is awesome @mathieuboudreau !

[mathieuboudreau]

Here's an example of now a whole-body merged label, with very careful consideration to the order of setting the labels.

Video: https://drive.google.com/file/d/1P3lodhJuEGwOD8bIalQVok8NH4jt2hGx/view?usp=share_link
NIfTI file: https://drive.google.com/file/d/1XShCz7_1YTLeYOldPK54Go7ouY2WQsEn/view?usp=share_link

Some tissue islands related to SAMSEG could still be removed & brain should be smoothed

[mathieuboudreau]

One thing I just noticed is that, using my original label order, part of the sinuses got cropped out,

(left - add skull first then sinus; right - add sinus first then skull)

But when doing it the correct way, some of the sinus pertrudes the skull, i.e.

So either the skull needs to increase by 1-2 mm, or the sinus needs to shrink by that much

[mathieuboudreau]

Here's about the best I've gotten so far; I've added a small smoothing/island removal/2 mm shrink of the sinus and 2 mm shrink of the ear canal, 3 mm expansion of the body (to not create a gap between the body and sinus), and a island removal and smoothing of the brain.

3D video: https://drive.google.com/file/d/1_NVWvttd9QE3DvI_MsZwuWFIsgljySke/view?usp=share_link
2D video: https://drive.google.com/file/d/1BFsmSk8Ls1b6rB7OnAvVuV0vSfFd3lp6/view?usp=share_link
NIfTI file: https://drive.google.com/file/d/1Vujk1BIgcN-paA8BokCFXqlrRgZXFWaq/view?usp=share_link

Maybe this is the point where it would be a good idea to convert this tissue segmentation to a chi distribution, followed by the B0 map? Before I run this on all the remaining subjects, and polish the code/scripts.

One final thing I notice, which I can't fix, is there seems to be a mis-alignement between the vertebraes and canal at the extermeties of the spinal cord, not sure why (were different tools used to segment both?):

[mathieuboudreau]

One final thing I notice, which I can't fix, is there seems to be a mis-alignement between the vertebraes and canal at the extermeties of the spinal cord, not sure why (were different tools used to segment both?):

Actually, I had the idea to double check the nifti affine's, and they're the same for all the segment files except for the canal file:

Volume loaded :data/sub-amuAL_T1w_label-canal_seg.nii.gz
Affine is:
[[ 9.99812543e-01  1.00949360e-16  1.93620156e-02 -9.23610458e+01]
 [-9.99812545e-17  1.00000000e+00 -5.09681033e-17 -1.41025833e+02]
 [-1.93620156e-02  4.90227068e-17  9.99812543e-01 -5.67712158e+02]
 [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]
Volume loaded :data/sub-amuAL_T1w_label-spine_dseg.nii.gz
Affine is:
[[-9.99812543e-01  1.00949360e-16  1.93620156e-02  7.96067123e+01]
 [ 9.99812545e-17  1.00000000e+00 -5.09681033e-17 -1.41025833e+02]
 [ 1.93620156e-02  4.90227068e-17  9.99812543e-01 -5.71042419e+02]
 [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]

1. why is this?
2. I can fix this prior to merging the segmentations, brb

[mathieuboudreau]

Here's the file with fixed orientation: https://drive.google.com/file/d/14OODt4970AMxGHKABx4x5TDPTDuQTTIl/view?usp=sharing

2D video: https://drive.google.com/file/d/1E6NugQNs178XHFbPBXdW_KgUHWN7uD_M/view?usp=sharing

Screenshots:

[jcohenadad]

#4 (comment) this post could make a fantastic figure for the ISMRM abstract, showing the structures being added iteratively. @mathieuboudreau maybe just keep a note of it once we will put the abstract together (ie: after sending the data to MGH, whcih is priority)

[mathieuboudreau]

#4 (comment) this post could make a fantastic figure for the ISMRM abstract, showing the structures being added iteratively. @mathieuboudreau maybe just keep a note of it once we will put the abstract together (ie: after sending the data to MGH, whcih is priority)

Maybe something like this?

[mathieuboudreau]

Refactored the scripts a bit and filled in some gaps to help run it on all the subjects, and tested it now on a second subject:

NIfTI file: https://drive.google.com/file/d/1wk3TivLVpcj-2C1XrHgRAUGkYvWBhE1h/view?usp=share_link

Tissue masks:

This subject's SAMSEG segmentation of the head was cropped a bit more than the first one I tried, leading to more disconnect between the spinal canal and the brain, and cropped eyes.

I guess now there are broad next steps?

• Generate the chi maps then B0 maps for a few test subjects
  • For chi2b0, how/who? -> https://github.com/shimming-toolbox/susceptibility-to-fieldmap-fft
• Run the tissue merging script on all subjects
• Add the merged labels to git-annex + upload entire dataset to openneuro

Wha exactly does MGH need - the B0 maps or just the tissue segmentations / chi maps?

[mathieuboudreau]

Here's a field distortion / B0 map generated from https://github.com/shimming-toolbox/susceptibility-to-fieldmap-fft on the first subject from above (sub-amuA),

Using chi = 8.86 for bone (from a reference I found, Sumanaweera, T. S., Glover, G. H., Binford, T. O., & Adler, J. R. (1993). MR susceptibility misregistration correction. IEEE Transactions on Medical Imaging, 12(2), 251–259. doi:10.1109/42.232253)

Using chi = -11.5 for bone (the other value mentioned in https://github.com/shimming-toolbox/tissue-to-MRproperty/blob/bd3d46ed5d69f3a0401ab38263f769363ed988e7/functions/utils/select_tool.py#L259, from an unknown reference)

I find it odd that the values are so much lower directly above the head (below blue, so black) vs in the axial plane outside the head where it's higher ( reaching positive numbers, red), this doesn't make sense to me conceptually from a physics point of view, but I haven't looked at the B0 simulation code

[sriosq]

@mathieuboudreau this is very interesting, right now we are investigating how different susceptibility values affect the fieldmaps. The value for bone was updated after @CharlesPageot did a literature research so he might be able to tell you were this value comes from. Also, in the tissue to MR property, every susceptibilit value is reference to free space so its important to check what it is referenced to. In the paper you mentioned they say its - 8.86 referenced to air.

[sriosq]

This is an important topic because i would also like to confirm the value of susceptibility for cartilage @CharlesPageot. Thanks!

[mathieuboudreau]

@mathieuboudreau this is very interesting, right now we are investigating how different susceptibility values affect the fieldmaps. The value for bone was updated after @CharlesPageot did a literature research so he might be able to tell you were this value comes from. Also, in the tissue to MR property, every susceptibilit value is reference to free space so its important to check what it is referenced to. In the paper you mentioned they say its - 8.86 referenced to air.

Good point! I was playing around with sifferent values because i saw two different ones in the dictionnary, shimming-toolbox/tissue-to-MRproperty#15, and couldnt find any references or explanation for this in this code, shimming-toolbox/tissue-to-MRproperty#14. Thats why i started looking for values in references, but itll be a quick update once I know the final chi values to use for each tissue!

[CharlesPageot]

The -11.5 for bone comes from this abstract. In this study, they found a bone susceptibility of -2.46 relative to water. For cartillage @sriosq, i had trouble finding other sources, so I statyed with the value in the Matlab repo which already had a reference.

[jcohenadad]

This subject's SAMSEG segmentation of the head was cropped a bit more than the first one I tried, leading to more disconnect between the spinal canal and the brain, and cropped eyes.

Indeed, we will need to deal with this after the ISMRM deadline (maybe we can open a specific issue for this)

I guess now there are broad next steps?
Generate the chi maps then B0 maps for a few test subjects
For chi2b0, how/who? -> https://github.com/shimming-toolbox/susceptibility-to-fieldmap-fft
Run the tissue merging script on all subjects
Add the merged labels to git-annex + upload entire dataset to openneuro

Yes! For openneuro, I'm still waiting for the 'OK' from Marseille, but all the other tasks are indeed needed

What exactly does MGH need - the B0 maps or just the tissue segmentations / chi maps?

@Govish can clarify

[mathieuboudreau]

I've run the label smoothing & merging script on all subject, converted this merged label map to chi with some values @sriosq suggested to use,

and run the chi-to-b0 map software on all subjects, but refactoring so that the padding uses numpy's np.pad function + the edge option (repeats the values at each edge for the padding thickness, i.e. air above and to the side of the head/torso will just be extended to be more air, and the bottom of the torso will be repeated to avoid an unrealistic sharp torso-to-air interface that would be introduced by zero-padding). For this, see this branch: shimming-toolbox/susceptibility-to-fieldmap-fft@4463dbd

Here's a GIF of the fieldmap for all subjects (may be slow to load)

The scale goes from -5 to 0, -5 being blue, -2.5 white, 0 red.

Some subjects have blank fieldmaps; the output from the chi-to-b0 software were NaN's, not sure why yet. I also discarded two subjects (unfErssm001 and unfErssm021), as as 001 was missing the brain SAMSEG labels, and 021 was missing the T1w magnitude images.

[mathieuboudreau]

@Govish, are the fieldmaps in the animation in my comment above (gif, may take a few seconds to load) closer to what you were looking for?

[Govish]

To my untrained eye they look pretty good! I'll check with Jason to see if they seem reasonable to him too.
Most of our challenges came from the mask, and it looks like you guys are making great progress on that front too!

[jcohenadad]

about the padding issue, @mathieuboudreau this figure from [Shang Y, Theilenberg S, Terekhov M, Mattar W, Peng B, Jambawalikar SR, Schreiber LM, Juchem C. High-resolution simulation of B0 field conditions in the human heart from segmented computed tomography images. NMR Biomed. 2022 Aug;35(8):e4739.] is possibly of relevance:

[mathieuboudreau]

@jcohenadad thanks! I went with edge padding instead of zero padding, since our volume is cut off at the torso unlike them, and that would create a ficticious air-tissue boundary at the torso. Also we have chi outside the body set to air, so zero padding would create another interface

[mathieuboudreau]

But, i did notice the effects of padding width to zeroth order B0 values like they seem to have observed

[mathieuboudreau]

RE: the topic of this original issue title (bidsifying the dataset), I just finished pushing all the smoothed labels, chi maps, and simulated B0 map to the data.neuro.polymtl.ca in the mb/air-tissue-labels branch of the whole-spine dataset.

In addition, I setup my pipelines/functions to automatically generate detailed json-sidecars for each produced NIfTI file after they are saved, hopefully enough so that 1) they can be reproduced, 2) they can be traced back to different versions of the softwares/code used, and 3) provide enough information so that they can be used (i.e. details about what the labels mean, what anatomy was assigned which chi value, etc).

Here are some examples for different types of files for one subject:

Simulated B0 map: sub-amuALT_T1w_fmap_b0-sim.json

{
	"author":"mathieuboudreau",
	"creation date":"2024-10-31 21:19:48",
	"script":"compute_fieldmap",
	"script source":"https://github.com/shimming-toolbox/susceptibility-to-fieldmap-fft",
	"script commit hash":"d9f785b082fb145d547ff03ae53f23f1564ccc38",
	"input file":"/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/sub-amuALT/anat/sub-amuALT_T1w-chi.nii.gz",
	"padding":50,
	"command":"compute_fieldmap -i /Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/sub-amuALT/anat/sub-amuALT_T1w-chi.nii.gz -o /Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/sub-amuALT/fmap/sub-amuALT_T1w_fmap_b0-sim.nii.gz -b 50 -m b0SimISMRM"
}

Chi map: sub-amuALT_T1w-chi.json

{
    "author": "mathieuboudreau",
    "date": "2024-10-31 21:15:40",
    "script": "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/b0-fieldmap-realistic-simulation/b0realsim/label_to_chi.py",
    "script source": "https://github.com/shimming-toolbox/b0-fieldmap-realistic-simulation",
    "script commit hash": "30d3e91b5071b6c41c97cfd9abf7b339365e4464",
    "input file": "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-all.nii.gz",
    "command": "python label_to_chi.py -s /Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/sub-amuALT",
    "anatomy": {
        "background": {
            "label": 0,
            "chi": 0.35
        },
        "body": {
            "label": "1",
            "chi": -9.05
        },
        "sinus": {
            "label": 2,
            "chi": -2
        },
        "earcanal": {
            "label": 3,
            "chi": -2
        },
        "trachea": {
            "label": 4,
            "chi": -4.2
        },
        "rightlung": {
            "label": 5,
            "chi": -4.2
        },
        "leftlung": {
            "label": 6,
            "chi": -4.2
        },
        "brain": {
            "label": 56,
            "chi": -9.04
        },
        "eyes": {
            "label": 60,
            "chi": -9.05
        },
        "skull": {
            "label": 91,
            "chi": -11
        },
        "verterbae": {
            "label": 92,
            "chi": -11
        },
        "disks": {
            "label": 93,
            "chi": -9.055
        },
        "canal": {
            "label": 100,
            "chi": -9.055
        }
    }
}

Combined labels for all anatomy: sub-amuALT_T1w_label-all.json

{
    "author": "mathieuboudreau",
    "date": "2024-10-31 12:08:01",
    "script": "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/b0-fieldmap-realistic-simulation/b0realsim/merge_labels.py",
    "script source": "https://github.com/shimming-toolbox/b0-fieldmap-realistic-simulation",
    "script commit hash": "30d3e91b5071b6c41c97cfd9abf7b339365e4464",
    "input files": [
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-air_tissue.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-canal_seg.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-spine_dseg.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-brain.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-skull.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-eyes.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-skin.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-sinus.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-earcanal.nii.gz",
        "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-body.nii.gz"
    ],
    "anatomy": {
        "background": {
            "label": 0
        },
        "body": {
            "label": "1"
        },
        "sinus": {
            "label": 2
        },
        "earcanal": {
            "label": 3
        },
        "trachea": {
            "label": 4
        },
        "rightlung": {
            "label": 5
        },
        "leftlung": {
            "label": 6
        },
        "brain": {
            "label": 56
        },
        "eyes": {
            "label": 60
        },
        "skull": {
            "label": 91
        },
        "verterbae": {
            "label": 92
        },
        "disks": {
            "label": 93
        },
        "canal": {
            "label": 100
        }
    },
    "command": "python merge_labels.py -s /Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/sub-amuALT"
}

Anatomical (skull) segmentation after smoothing using Slicer: sub-amuALT_T1w_label-skull.json

{
    "author": "mathieuboudreau",
    "date": "2024-10-31 12:07:31",
    "script": "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/b0-fieldmap-realistic-simulation/b0realsim/slicer_scripts/smooth_segment.py",
    "script source": "https://github.com/shimming-toolbox/b0-fieldmap-realistic-simulation",
    "script commit hash": "30d3e91b5071b6c41c97cfd9abf7b339365e4464",
    "input file": "/Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-skull-raw.nii.gz",
    "label": {
        "anatomy": "skull",
        "value": 1,
        "input file value": 1,
        "processing steps": {
            "islands": {
                "operation": "REMOVE_SMALL_ISLANDS"
            },
            "smoothing": {
                "method": "MORPHOLOGICAL_CLOSING",
                "kernel_size_mm": 15
            },
            "grow": {
                "margin_mm": 3
            },
            "shrink": {
                "margin_mm": 2
            }
        }
    },
    "command": "/Applications/Slicer.app/Contents/MacOS/Slicer --python-script /Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/b0-fieldmap-realistic-simulation/b0realsim/slicer_scripts/smooth_segment.py -m /Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/sub-amuALT/anat/sub-amuALT_T1w.nii.gz -s /Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-skull-raw.nii.gz -o /Users/mathieuboudreau/neuropoly/projects/shimming-toolbox/data/data.neuro.polymtl.ca/whole-spine/derivatives/labels/sub-amuALT/anat/sub-amuALT_T1w_label-skull.nii.gz -a skull",
    "slicer version": "5.6",
    "slicer repository revision": "f10cd8c"
}

Writing these sidecars were done a bit differently depending on if a shell script or python script generated them; for those interested, here's some of the example code (could be cleaned up and refactored, but time is running short for the ISMRM deadline):

Shell script:

b0-fieldmap-realistic-simulation/b0realsim/slicer_scripts/run_all_b0.sh

Lines 39 to 46 in 30d3e91

	# Define the JSON format string
	JSON_FMT='{\n\t\\\"author\\\":\\\"%s\\\",\n\t\\\"creation date\\\":\\"%s",\n\t\\\"script\\\":\\\"%s\\\",\n\t\\\"script source\\\":\\\"%s\\\",\n\t\\\"script commit hash\\\":\\\"%s\\\",\n\t\\\"input file\\\":\\\"%s\\\",\n\t\\\"padding\\\":%s,\n\t\\\"command\\\":\\\"%s\\\"\n}\n'
	# Create the JSON string with variables replaced by their values, except for the date
	JSON_STR=$(printf "$JSON_FMT" "$USER" "\$(date +\"%Y-%m-%d %H:%M:%S\")\\" "$SCRIPT_NAME" "$SCRIPT_SOURCE" "$SCRIPT_COMMIT_HASH" "$INPUT_FILE" "$PADDING" "$COMMAND")
	# Write the command to the script file
	echo "echo \"$JSON_STR\" >| $OUTPUT_SIDECAR" >> compute_fieldmaps.sh

Python script:

b0-fieldmap-realistic-simulation/b0realsim/slicer_scripts/smooth_segment.py

Lines 251 to 277 in 30d3e91

	# JSON sidecar
	repo = git.Repo(search_parent_directories=True)
	bids_sidecar = {}
	bids_sidecar['author'] = os.getenv('USER')
	bids_sidecar['date'] = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
	bids_sidecar['script'] = str(Path(os.path.abspath(__file__)).resolve())
	bids_sidecar['script source'] = repo.remotes.origin.url
	bids_sidecar['script commit hash'] = repo.head.object.hexsha
	bids_sidecar["input file"] = str(Path(segmentation_path).resolve())
	bids_sidecar["label"] = {}
	bids_sidecar["label"]["anatomy"] = anatomy
	bids_sidecar["label"]["value"] = 1
	bids_sidecar["label"]["input file value"] = int(segment_id.split("_")[1])
	bids_sidecar["label"]["processing steps"] = processing_steps
	bids_sidecar['command'] = "/Applications/Slicer.app/Contents/MacOS/Slicer --python-script " + bids_sidecar['script'] + " -m " + main_volume_path + " -s " + segmentation_path + " -o " + output_path + " -a " + anatomy
	bids_sidecar["slicer version"] = str(slicer.app.majorVersion) + "." + str(slicer.app.minorVersion)
	bids_sidecar["slicer repository revision"] = str(slicer.app.repositoryRevision)
	json_file = str(Path(output_path).resolve()).replace(".nii.gz", ".json")
	if os.path.exists(json_file):
	os.remove(json_file)
	with open(json_file, 'w', encoding='utf-8') as f:
	json.dump(bids_sidecar, f, ensure_ascii=False, indent=4)