Merge pull request #62 from neuropoly/nb/issues_59_and_41

Fixing issues related to output file management (#57), documentation (#58), QC procedure (#59, #61), template max range (#41, #60)
neuropoly · Aug 7, 2023 · 5f511c7 · 5f511c7
2 parents 1244bbd + cd8b936
commit 5f511c7
Show file tree

Hide file tree

Showing 4 changed files with 74 additions and 54 deletions.
diff --git a/README.md b/README.md
@@ -11,7 +11,7 @@ Framework for creating MRI templates of the spinal cord. The framework has two d
 ### [Spinal Cord Toolbox (SCT)](https://spinalcordtoolbox.com/)
 
 Installation instructions can be found [here](https://spinalcordtoolbox.com/user_section/installation.html).
-For the following repository, we used SCT in developper mode (commit `e740edf4c8408ffa44ef7ba23ad068c6d07e4b87`).
+For the following repository, we used SCT in developper mode (commit `49a40673e6d1521eb7c2d1d6d7b338ab6811448d`).
 
 ### [ANIMAL registration framework](https://github.com/vfonov/nist_mni_pipelines)
 
@@ -62,28 +62,28 @@ dataset/
     └── labels
         └── sub-03
             └── anat
-                └── sub-03_T1w_label-SC_seg.nii.gz  <-- Spinal cord segmentation; `_T1w` can be replaced by the value of `suffix_image` in configuration.json
-                └── sub-03_T1w_label-disc.nii.gz  <---- Disc labels; `_T1w` can be replaced by the value of `suffix_image` in configuration.json
-                └── sub-03_T2w_label-SC_seg.nii.gz
-                └── sub-03_T2w_label-disc.nii.gz
+                └── sub-03_T1w_label-SC_mask.nii.gz  <-- Spinal cord segmentation; `_T1w` can be replaced by the value of `suffix_image` in configuration.json
+                └── sub-03_T1w_labels-disc.nii.gz  <---- Disc labels; `_T1w` can be replaced by the value of `suffix_image` in configuration.json
+                └── sub-03_T2w_label-SC_mask.nii.gz
+                └── sub-03_T2w_labels-disc.nii.gz
 ```
 
 
 ## Step 1. Data preprocessing
 
 This pipeline includes the following steps:
-1. Install SCT
-2. Edit configuration file
-3. Segment spinal cord and vertebral discs
-4. Quality control (QC) segmentation and labels using SCT's web-based HTML QC report, and download YML files of data to be corrected
-5. Manually correct files when correction is needed using https://github.com/spinalcordtoolbox/manual-correction
-6. Copy the non-corrected and corrected files back in the input dataset
-7. Normalize spinal cord across subjects
+1. Install SCT;
+2. Edit configuration file;
+3. Segment spinal cord and vertebral discs;
+4. Quality control (QC) segmentation and labels using SCT's web-based HTML QC report, and download YML files of data to be corrected;
+5. Manually correct files when correction is needed using the [SCT manual correction](https://spinalcordtoolbox.com/user_section/tutorials/registration-to-template/vertebral-labeling.html) repository;
+6. Copy the non-corrected and corrected files back in the input dataset;
+7. Normalize spinal cord across subjects;
 8. Quality control (QC) spinal cord normalization across subjects.
 
 ### 1.1 Install SCT
 
-SCT is used for all preprocessing steps. The current version of the pipeline uses SCT development version (commit `e740edf4c8408ffa44ef7ba23ad068c6d07e4b87`) as we prepare for the release of SCT 6.0.
+SCT is used for all preprocessing steps. The current version of the pipeline uses SCT development version (commit `49a40673e6d1521eb7c2d1d6d7b338ab6811448d`) as we prepare for the release of SCT 6.0.
 
 Once SCT is installed, make sure to activate SCT's virtual environment because the pipeline will use SCT's API functions.
 
@@ -126,11 +126,30 @@ With:
 
 ### 1.4 Quality control (QC) labels
 
-* Spinal cord segmentation (or centerlines) and disc labels can be displayed by opening: `/PATH_OUT/qc/index.html`
-* See [tutorial](https://spinalcordtoolbox.com/user_section/tutorials/registration-to-template/vertebral-labeling.html) for tips on how to QC and fix segmentation (or centerline) and/or disc labels manually.
+* Spinal cord segmentation (or centerlines) and disc labels can be displayed by opening: `PATH_OUT/qc/index.html`;
+* Quality control (QC) segmentation and labels using [SCT's web-based HTML QC report](https://spinalcordtoolbox.com/overview/concepts/inspecting-results-qc-fsleyes.html#how-do-i-use-the-qc-report), and download YML files (`qc_fail.yml`) of data to be corrected.
 
+### 1.5 Manual correction
 
-### 1.5 Normalize spinal cord across subjects
+Manually correct files when correction is needed, following the [SCT manual correction](https://github.com/spinalcordtoolbox/manual-correction) repository:
+* Installation of `manual-correction`
+* `manual_correction.py` script:
+```
+python manual_correction.py -path-img PATH_OUT/data_processed -suffix-files-seg '_label-SC_mask' -suffix-files-label '_labels-disc' -config path/to/qc_fail.yml
+```
+* `copy_files_to_derivatives.py` script:
+```
+python copy_files_to_derivatives.py -path-in PATH_OUT/data_processed/derivatives/labels -path-out PATH_DATA/derivatives/labels
+```
+
+> **Note**
+- `PATH_DATA`: from `configuration.json` absolute path to the input [BIDS dataset](#dataset-structure).
+- `PATH_OUT`: The location where to output the processed data, results, the logs and the QC information. Example: `/scratch/template_preproc_YYYYMMDD-HHMMSS`. Used in Step 1.3.
+
+> **Note**
+> See [tutorial](https://spinalcordtoolbox.com/user_section/tutorials/registration-to-template/vertebral-labeling.html) for tips on how to QC and fix segmentation (or centerline) and/or disc labels manually.
+
+### 1.6 Normalize spinal cord across subjects
 
 `preprocess_normalize.py` contains several functions to normalize the spinal cord across subjects, in preparation for template generation. More specifically:
 * Extracting the spinal cord centerline and computing the vertebral distribution along the spinal cord, for all subjects.
@@ -144,7 +163,7 @@ Run:
 python preprocess_normalize.py configuration.json
 ```
 
-### 1.6 Quality control (QC) spinal cord normalizatio across subjects
+### 1.7 QC of spinal cord normalization
 
 One the preprocessing is performed, please check your data. The preprocessing results should be a series of straight images registered in the same space, with all the vertebral levels aligned with each others.
 

diff --git a/configuration_default.json b/configuration_default.json
@@ -4,6 +4,5 @@
 	"data_type": "anat",
 	"contrast": "t1",
 	"suffix_image": "_T1w",
-	"first_disc": "1",
 	"last_disc": "26"
 }
diff --git a/preprocess_normalize.py b/preprocess_normalize.py
@@ -21,8 +21,8 @@
     └── labels
         ├── sub-XXX
         │   └── anat
-        │       │──sub-XXX_T1w_label-SC_seg.nii.gz  <---- spinal cord segmentation
-        │       └──sub-XXX_T1w_label-SC_seg_labeled_discs.nii.gz  <---- disc labels
+        │       │──sub-XXX_T1w_label-SC_mask.nii.gz  <---- spinal cord segmentation
+        │       └──sub-XXX_T1w_labels-disc.nii.gz  <---- disc labels
         ...
 
 Usage: `python preprocess_normalize.py configuration.json`
@@ -100,7 +100,7 @@ def read_dataset(fname_json = 'configuration.json', path_data = './'):
     with open(fname_json) as data_file: dataset_info = json.load(data_file)
 
     error = ''
-    key_list = ["path_data", "include-list", "data_type", "contrast", "suffix_image", "first_disc", "last_disc"]
+    key_list = ["path_data", "include-list", "data_type", "contrast", "suffix_image", "last_disc"]
 
     for key in key_list:
         if key not in dataset_info.keys(): error += 'Dataset configuration file ' + fname_json + ' must contain the field ' + key + '.\n'
@@ -130,8 +130,8 @@ def generate_centerline(dataset_info, algo_fitting = 'linear', smooth = 50, degr
     # obtaining centerline of each subject
     for subject_name in list_subjects:
         fname_image = path_data + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '.nii.gz'
-        fname_image_seg = path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-SC_seg.nii.gz'
-        fname_image_discs = path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-disc.nii.gz'
+        fname_image_seg = path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-SC_mask.nii.gz'
+        fname_image_discs = path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_labels-disc.nii.gz'
 
         if os.path.isfile(fname_image_seg):
             print(subject_name + ' SC segmentation exists. Extracting centerline from ' + fname_image_seg)
@@ -204,18 +204,18 @@ def average_centerline(list_centerline, dataset_info, use_ICBM152 = False, use_l
              position_template_discs: index of intervertebral discs along the template centerline
     """
 
-    last_disc = int(dataset_info['last_disc'])
     # extracting centerline from ICBM152
     if use_ICBM152: centerline_icbm152 = compute_ICBM152_centerline(dataset_info)
 
+    last_disc = int(dataset_info['last_disc'])
     list_dist_discs = []
     for centerline in list_centerline: list_dist_discs.append(centerline.distance_from_C1label)
 
     # generating custom list of average vertebral lengths
     new_vert_length = {}
     for dist_discs in list_dist_discs:
         for i, disc_label in enumerate(dist_discs):
-            if i <= last_disc:
+            if (i + 1) <= last_disc:
                 if disc_label == 'PMJ':
                     length = abs(dist_discs[disc_label] - dist_discs['PMG'])
                 elif disc_label == 'PMG':
@@ -268,10 +268,10 @@ def average_centerline(list_centerline, dataset_info, use_ICBM152 = False, use_l
         distances_discs_from_C1['PMG'] = -average_length['PMG'][1]
         if 'PMJ' in average_length:
             distances_discs_from_C1['PMJ'] = -average_length['PMG'][1] - average_length['PMJ'][1]
-    for disc_number in Centerline.potential_list_labels:
-        if disc_number not in [50, 49, 1] and Centerline.regions_labels[disc_number] in average_length:
-            distances_discs_from_C1[Centerline.regions_labels[disc_number]] = distances_discs_from_C1[Centerline.regions_labels[disc_number - 1]] + average_length[Centerline.regions_labels[disc_number]][1]
-
+    for disc_number in range(last_disc + 1): #Centerline.potential_list_labels:
+        if disc_number not in [0, 1, 48, 50]: #and Centerline.regions_labels[disc_number] in average_length:
+            distances_discs_from_C1[Centerline.regions_labels[disc_number]] = distances_discs_from_C1[Centerline.regions_labels[disc_number - 1]] + average_length[Centerline.regions_labels[disc_number - 1]][1]
+    
     # calculating discs average distances from C1
     average_distances = []
     for disc_label in distances_discs_from_C1:
@@ -338,7 +338,7 @@ def average_centerline(list_centerline, dataset_info, use_ICBM152 = False, use_l
                 position_template_discs[disc] = coord_disc
     else:
         coord_ref = np.array([0.0, 0.0, 0.0])
-        for disc in average_length:
+        for disc in average_positions_from_C1:
             coord_disc = coord_ref.copy()
             coord_disc[2] -= average_positions_from_C1[disc] - average_positions_from_C1[label_ref]
             position_template_discs[disc] = coord_disc
@@ -437,15 +437,15 @@ def generate_initial_template_space(dataset_info, points_average_centerline, pos
         else:
             sct.printv(str(coord_pix))
             sct.printv('ERROR: the disc label ' + str(disc) + ' is not in the template image.')
-    image_discs.save(path_template + 'template_label-disc.nii.gz', dtype = 'uint8')
-    print(f'\nSaving disc positions in {image_discs.orientation} orientation as {path_template}template_label-disc.nii.gz\n')
+    image_discs.save(path_template + 'template_labels-disc.nii.gz', dtype = 'uint8')
+    print(f'\nSaving disc positions in {image_discs.orientation} orientation as {path_template}template_labels-disc.nii.gz\n')
 
     # generate template centerline as a npz file
     param_centerline = ParamCenterline(algo_fitting = algo_fitting, smooth = smooth, degree = degree, minmax = minmax) 
     # centerline params of original template centerline had options that you cannot just provide `get_centerline` with anymroe (algo_fitting = 'nurbs', nurbs_pts_number = 4000, all_slices = False, phys_coordinates = True, remove_outliers = True)
     _, arr_ctl, arr_ctl_der, _ = get_centerline(image_centerline, param = param_centerline, space = 'phys')
     centerline_template = Centerline(points_x = arr_ctl[0], points_y = arr_ctl[1], points_z = arr_ctl[2], deriv_x = arr_ctl_der[0], deriv_y = arr_ctl_der[1], deriv_z = arr_ctl_der[2])
-    centerline_template.compute_vertebral_distribution(coord_physical)        
+    centerline_template.compute_vertebral_distribution(coord_physical)
     centerline_template.save_centerline(fname_output = path_template + 'template_label-centerline')
     print(f'\nSaving template centerline as .npz file (saves all Centerline object information, not just coordinates) as {path_template}template_label-centerline.npz\n')
 
@@ -468,8 +468,8 @@ def straighten_all_subjects(dataset_info, normalized = False):
         if not os.path.exists(folder_out): os.makedirs(folder_out)
 
         fname_image = path_data + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '.nii.gz'
-        fname_image_seg = path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-SC_seg.nii.gz'
-        fname_image_discs = path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-disc.nii.gz'
+        fname_image_seg = path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-SC_mask.nii.gz'
+        fname_image_discs = path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_labels-disc.nii.gz'
         fname_image_centerline =  path_data + 'derivatives/labels/' + subject_name +  '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-centerline.nii.gz'
         fname_out = subject_name + dataset_info['suffix_image'] + '_straight_norm.nii.gz' if normalized else subject_name + dataset_info['suffix_image'] + '_straight.nii.gz' 
 
@@ -484,7 +484,7 @@ def straighten_all_subjects(dataset_info, normalized = False):
             ' -s ' + fname_input_seg + 
             ' -dest ' + path_template + 'template_label-centerline.nii.gz' + 
             ' -ldisc-input ' + fname_image_discs +        
-            ' -ldisc-dest ' + path_template + 'template_label-disc.nii.gz' + 
+            ' -ldisc-dest ' + path_template + 'template_labels-disc.nii.gz' + 
             ' -ofolder ' + folder_out + 
             ' -o ' + fname_out + 
             ' -disable-straight2curved' + 
@@ -666,7 +666,7 @@ def main(configuration_file):
     dataset_info = read_dataset(configuration_file)
 
     # generating centerlines
-    list_centerline = generate_centerline(dataset_info = dataset_info) 
+    list_centerline = generate_centerline(dataset_info = dataset_info)
 
     # computing average template centerline and vertebral distribution
     points_average_centerline, position_template_discs = average_centerline(list_centerline = list_centerline,

diff --git a/preprocess_segment.sh b/preprocess_segment.sh
@@ -34,8 +34,7 @@ PATH_DATA=$(echo "$json_data" | sed -n 's/.*"path_data": "\(.*\)".*/\1/p')
 DATA_TYPE=$(echo "$json_data" | sed -n 's/.*"data_type": "\(.*\)".*/\1/p')
 IMAGE_SUFFIX=$(echo "$json_data" | sed -n 's/.*"suffix_image": "\(.*\)".*/\1/p')
 CONTRAST=$(echo "$json_data" | sed -n 's/.*"contrast": "\(.*\)".*/\1/p')
-PATH_DATASET_OUTPUT="${PATH_DATA}derivatives/labels/${SUBJECT}/${DATA_TYPE}"
-mkdir -p ${PATH_DATASET_OUTPUT}
+FILE=$PATH_DATA_PROCESSED/$SUBJECT/$DATA_TYPE/${SUBJECT}${IMAGE_SUFFIX}.nii.gz
 
 # Uncomment for full verbose
 # set -v
@@ -57,58 +56,61 @@ start=`date +%s`
 sct_check_dependencies -short
 
 # Go to folder where data will be copied and processed
-cd $PATH_DATA_PROCESSED
+mkdir -p $PATH_DATA_PROCESSED/$SUBJECT/$DATA_TYPE
+mkdir -p $PATH_DATA_PROCESSED/derivatives/labels/$SUBJECT/$DATA_TYPE
+cd $PATH_DATA_PROCESSED/derivatives/labels/$SUBJECT/$DATA_TYPE
 
 # Copy source images
-rsync -avzh $PATH_DATA/$SUBJECT/$DATA_TYPE/* .
+rsync -avzh $PATH_DATA/$SUBJECT/$DATA_TYPE/${SUBJECT}${IMAGE_SUFFIX}.nii.gz $PATH_DATA_PROCESSED/$SUBJECT/$DATA_TYPE
 
 
 # Segment spinal cord (SC) if does not exist
 # ======================================================================================================================
 
-FILE="${SUBJECT}${IMAGE_SUFFIX}.nii.gz"
-FILESEG="${SUBJECT}${IMAGE_SUFFIX}_label-SC_seg.nii.gz"
+FILESEG="${SUBJECT}${IMAGE_SUFFIX}_label-SC_mask.nii.gz"
 
 echo "Looking for segmentation: ${FILESEG}"
-if [[ -e "${PATH_DATASET_OUTPUT}/${FILESEG}" ]]; then
+if [[ -e "${FILESEG}" ]]; then
   echo "Found! Using SC segmentation that exists."
-  sct_qc -i ${FILE} -s "${PATH_DATASET_OUTPUT}/${FILESEG}" -p sct_deepseg_sc -qc ${PATH_QC} -qc-subject ${SUBJECT}
+  sct_qc -i ${FILE} -s "${FILESEG}" -p sct_deepseg_sc -qc ${PATH_QC} -qc-subject ${SUBJECT}
 else
   echo "Not found. Proceeding with automatic segmentation."
   # Segment spinal cord
   sct_deepseg_sc -i ${FILE} -o ${FILESEG} -c ${CONTRAST} -qc ${PATH_QC} -qc-subject ${SUBJECT}
-  mv ${FILESEG} "${PATH_DATASET_OUTPUT}/${FILESEG}"
 fi
 
+
 # Label discs if do not exist
 # ======================================================================================================================
 
-FILELABEL="${SUBJECT}${IMAGE_SUFFIX}_label-disc.nii.gz"
+FILELABEL="${SUBJECT}${IMAGE_SUFFIX}_labels-disc.nii.gz"
 
 echo "Looking for disc labels: ${FILELABEL}"
-if [[ -e "${PATH_DATASET_OUTPUT}/${FILELABEL}" ]]; then
+if [[ -e "${FILELABEL}" ]]; then
   echo "Found! Using vertebral labels that exist."
-  sct_qc -i ${FILE} -s "${PATH_DATASET_OUTPUT}/${FILELABEL}" -p sct_label_vertebrae -qc ${PATH_QC} -qc-subject ${SUBJECT}
+  sct_qc -i ${FILE} -s "${FILELABEL}" -p sct_label_vertebrae -qc ${PATH_QC} -qc-subject ${SUBJECT}
 else
   echo "Not found. Proceeding with automatic labeling."
   # Generate labeled segmentation
-  sct_label_vertebrae -i ${FILE} -s "${PATH_DATASET_OUTPUT}/${FILESEG}" -ofolder ${PATH_DATASET_OUTPUT} -c ${CONTRAST} -qc "${PATH_QC}" -qc-subject "${SUBJECT}"
-  mv "${PATH_DATASET_OUTPUT}/${SUBJECT}${IMAGE_SUFFIX}_label-SC_seg_labeled_discs.nii.gz" "${PATH_DATASET_OUTPUT}/${FILELABEL}"
-  rm "${PATH_DATASET_OUTPUT}/${SUBJECT}${IMAGE_SUFFIX}_label-SC_seg_labeled.nii.gz"
+  sct_label_vertebrae -i ${FILE} -s "${FILESEG}" -c ${CONTRAST} -qc "${PATH_QC}" -qc-subject "${SUBJECT}"
+  mv "${SUBJECT}${IMAGE_SUFFIX}_label-SC_mask_labeled_discs.nii.gz" "${FILELABEL}"
+  rm "${SUBJECT}${IMAGE_SUFFIX}_label-SC_mask_labeled.nii.gz"
 fi
 
+
 # Verify presence of output files and write log file if error
 # ======================================================================================================================
 FILES_TO_CHECK=(
   "$FILESEG"
   "$FILELABEL"
 )
 for file in "${FILES_TO_CHECK[@]}"; do
-  if [ ! -e "${PATH_DATASET_OUTPUT}/${file}" ]; then
-    echo "${PATH_DATASET_OUTPUT}/${file} does not exist" >> "${PATH_LOG}/error.log"
+  if [ ! -e "${file}" ]; then
+    echo "${file} does not exist" >> "${PATH_LOG}/error.log"
   fi
 done
 
+
 # Display useful info for the log
 # ======================================================================================================================
 end=`date +%s`