diff --git a/README.md b/README.md index 397f398..b456929 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Framework for creating MRI templates of the spinal cord. The framework has two d ### [Spinal Cord Toolbox (SCT)](https://spinalcordtoolbox.com/) Installation instructions can be found [here](https://spinalcordtoolbox.com/user_section/installation.html). -For the following repository, we used SCT in developper mode (commit `e740edf4c8408ffa44ef7ba23ad068c6d07e4b87`). +For the following repository, we used SCT in developper mode (commit `49a40673e6d1521eb7c2d1d6d7b338ab6811448d`). ### [ANIMAL registration framework](https://github.com/vfonov/nist_mni_pipelines) @@ -62,28 +62,28 @@ dataset/ └── labels └── sub-03 └── anat - └── sub-03_T1w_label-SC_seg.nii.gz <-- Spinal cord segmentation; `_T1w` can be replaced by the value of `suffix_image` in configuration.json - └── sub-03_T1w_label-disc.nii.gz <---- Disc labels; `_T1w` can be replaced by the value of `suffix_image` in configuration.json - └── sub-03_T2w_label-SC_seg.nii.gz - └── sub-03_T2w_label-disc.nii.gz + └── sub-03_T1w_label-SC_mask.nii.gz <-- Spinal cord segmentation; `_T1w` can be replaced by the value of `suffix_image` in configuration.json + └── sub-03_T1w_labels-disc.nii.gz <---- Disc labels; `_T1w` can be replaced by the value of `suffix_image` in configuration.json + └── sub-03_T2w_label-SC_mask.nii.gz + └── sub-03_T2w_labels-disc.nii.gz ``` ## Step 1. Data preprocessing This pipeline includes the following steps: -1. Install SCT -2. Edit configuration file -3. Segment spinal cord and vertebral discs -4. Quality control (QC) segmentation and labels using SCT's web-based HTML QC report, and download YML files of data to be corrected -5. Manually correct files when correction is needed using https://github.com/spinalcordtoolbox/manual-correction -6. Copy the non-corrected and corrected files back in the input dataset -7. Normalize spinal cord across subjects +1. Install SCT; +2. Edit configuration file; +3. Segment spinal cord and vertebral discs; +4. Quality control (QC) segmentation and labels using SCT's web-based HTML QC report, and download YML files of data to be corrected; +5. Manually correct files when correction is needed using the [SCT manual correction](https://spinalcordtoolbox.com/user_section/tutorials/registration-to-template/vertebral-labeling.html) repository; +6. Copy the non-corrected and corrected files back in the input dataset; +7. Normalize spinal cord across subjects; 8. Quality control (QC) spinal cord normalization across subjects. ### 1.1 Install SCT -SCT is used for all preprocessing steps. The current version of the pipeline uses SCT development version (commit `e740edf4c8408ffa44ef7ba23ad068c6d07e4b87`) as we prepare for the release of SCT 6.0. +SCT is used for all preprocessing steps. The current version of the pipeline uses SCT development version (commit `49a40673e6d1521eb7c2d1d6d7b338ab6811448d`) as we prepare for the release of SCT 6.0. Once SCT is installed, make sure to activate SCT's virtual environment because the pipeline will use SCT's API functions. @@ -126,11 +126,30 @@ With: ### 1.4 Quality control (QC) labels -* Spinal cord segmentation (or centerlines) and disc labels can be displayed by opening: `/PATH_OUT/qc/index.html` -* See [tutorial](https://spinalcordtoolbox.com/user_section/tutorials/registration-to-template/vertebral-labeling.html) for tips on how to QC and fix segmentation (or centerline) and/or disc labels manually. +* Spinal cord segmentation (or centerlines) and disc labels can be displayed by opening: `PATH_OUT/qc/index.html`; +* Quality control (QC) segmentation and labels using [SCT's web-based HTML QC report](https://spinalcordtoolbox.com/overview/concepts/inspecting-results-qc-fsleyes.html#how-do-i-use-the-qc-report), and download YML files (`qc_fail.yml`) of data to be corrected. +### 1.5 Manual correction -### 1.5 Normalize spinal cord across subjects +Manually correct files when correction is needed, following the [SCT manual correction](https://github.com/spinalcordtoolbox/manual-correction) repository: +* Installation of `manual-correction` +* `manual_correction.py` script: +``` +python manual_correction.py -path-img PATH_OUT/data_processed -suffix-files-seg '_label-SC_mask' -suffix-files-label '_labels-disc' -config path/to/qc_fail.yml +``` +* `copy_files_to_derivatives.py` script: +``` +python copy_files_to_derivatives.py -path-in PATH_OUT/data_processed/derivatives/labels -path-out PATH_DATA/derivatives/labels +``` + +> **Note** +- `PATH_DATA`: from `configuration.json` absolute path to the input [BIDS dataset](#dataset-structure). +- `PATH_OUT`: The location where to output the processed data, results, the logs and the QC information. Example: `/scratch/template_preproc_YYYYMMDD-HHMMSS`. Used in Step 1.3. + +> **Note** +> See [tutorial](https://spinalcordtoolbox.com/user_section/tutorials/registration-to-template/vertebral-labeling.html) for tips on how to QC and fix segmentation (or centerline) and/or disc labels manually. + +### 1.6 Normalize spinal cord across subjects `preprocess_normalize.py` contains several functions to normalize the spinal cord across subjects, in preparation for template generation. More specifically: * Extracting the spinal cord centerline and computing the vertebral distribution along the spinal cord, for all subjects. @@ -144,7 +163,7 @@ Run: python preprocess_normalize.py configuration.json ``` -### 1.6 Quality control (QC) spinal cord normalizatio across subjects +### 1.7 QC of spinal cord normalization One the preprocessing is performed, please check your data. The preprocessing results should be a series of straight images registered in the same space, with all the vertebral levels aligned with each others. diff --git a/configuration_default.json b/configuration_default.json index ead1275..6322978 100644 --- a/configuration_default.json +++ b/configuration_default.json @@ -4,6 +4,5 @@ "data_type": "anat", "contrast": "t1", "suffix_image": "_T1w", - "first_disc": "1", "last_disc": "26" } \ No newline at end of file diff --git a/preprocess_normalize.py b/preprocess_normalize.py index baa329a..f743c24 100644 --- a/preprocess_normalize.py +++ b/preprocess_normalize.py @@ -21,8 +21,8 @@ └── labels ├── sub-XXX │ └── anat - │ │──sub-XXX_T1w_label-SC_seg.nii.gz <---- spinal cord segmentation - │ └──sub-XXX_T1w_label-SC_seg_labeled_discs.nii.gz <---- disc labels + │ │──sub-XXX_T1w_label-SC_mask.nii.gz <---- spinal cord segmentation + │ └──sub-XXX_T1w_labels-disc.nii.gz <---- disc labels ... Usage: `python preprocess_normalize.py configuration.json` @@ -100,7 +100,7 @@ def read_dataset(fname_json = 'configuration.json', path_data = './'): with open(fname_json) as data_file: dataset_info = json.load(data_file) error = '' - key_list = ["path_data", "include-list", "data_type", "contrast", "suffix_image", "first_disc", "last_disc"] + key_list = ["path_data", "include-list", "data_type", "contrast", "suffix_image", "last_disc"] for key in key_list: if key not in dataset_info.keys(): error += 'Dataset configuration file ' + fname_json + ' must contain the field ' + key + '.\n' @@ -130,8 +130,8 @@ def generate_centerline(dataset_info, algo_fitting = 'linear', smooth = 50, degr # obtaining centerline of each subject for subject_name in list_subjects: fname_image = path_data + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '.nii.gz' - fname_image_seg = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-SC_seg.nii.gz' - fname_image_discs = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-disc.nii.gz' + fname_image_seg = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-SC_mask.nii.gz' + fname_image_discs = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_labels-disc.nii.gz' if os.path.isfile(fname_image_seg): print(subject_name + ' SC segmentation exists. Extracting centerline from ' + fname_image_seg) @@ -204,10 +204,10 @@ def average_centerline(list_centerline, dataset_info, use_ICBM152 = False, use_l position_template_discs: index of intervertebral discs along the template centerline """ - last_disc = int(dataset_info['last_disc']) # extracting centerline from ICBM152 if use_ICBM152: centerline_icbm152 = compute_ICBM152_centerline(dataset_info) + last_disc = int(dataset_info['last_disc']) list_dist_discs = [] for centerline in list_centerline: list_dist_discs.append(centerline.distance_from_C1label) @@ -215,7 +215,7 @@ def average_centerline(list_centerline, dataset_info, use_ICBM152 = False, use_l new_vert_length = {} for dist_discs in list_dist_discs: for i, disc_label in enumerate(dist_discs): - if i <= last_disc: + if (i + 1) <= last_disc: if disc_label == 'PMJ': length = abs(dist_discs[disc_label] - dist_discs['PMG']) elif disc_label == 'PMG': @@ -268,10 +268,10 @@ def average_centerline(list_centerline, dataset_info, use_ICBM152 = False, use_l distances_discs_from_C1['PMG'] = -average_length['PMG'][1] if 'PMJ' in average_length: distances_discs_from_C1['PMJ'] = -average_length['PMG'][1] - average_length['PMJ'][1] - for disc_number in Centerline.potential_list_labels: - if disc_number not in [50, 49, 1] and Centerline.regions_labels[disc_number] in average_length: - distances_discs_from_C1[Centerline.regions_labels[disc_number]] = distances_discs_from_C1[Centerline.regions_labels[disc_number - 1]] + average_length[Centerline.regions_labels[disc_number]][1] - + for disc_number in range(last_disc + 1): #Centerline.potential_list_labels: + if disc_number not in [0, 1, 48, 50]: #and Centerline.regions_labels[disc_number] in average_length: + distances_discs_from_C1[Centerline.regions_labels[disc_number]] = distances_discs_from_C1[Centerline.regions_labels[disc_number - 1]] + average_length[Centerline.regions_labels[disc_number - 1]][1] + # calculating discs average distances from C1 average_distances = [] for disc_label in distances_discs_from_C1: @@ -338,7 +338,7 @@ def average_centerline(list_centerline, dataset_info, use_ICBM152 = False, use_l position_template_discs[disc] = coord_disc else: coord_ref = np.array([0.0, 0.0, 0.0]) - for disc in average_length: + for disc in average_positions_from_C1: coord_disc = coord_ref.copy() coord_disc[2] -= average_positions_from_C1[disc] - average_positions_from_C1[label_ref] position_template_discs[disc] = coord_disc @@ -437,15 +437,15 @@ def generate_initial_template_space(dataset_info, points_average_centerline, pos else: sct.printv(str(coord_pix)) sct.printv('ERROR: the disc label ' + str(disc) + ' is not in the template image.') - image_discs.save(path_template + 'template_label-disc.nii.gz', dtype = 'uint8') - print(f'\nSaving disc positions in {image_discs.orientation} orientation as {path_template}template_label-disc.nii.gz\n') + image_discs.save(path_template + 'template_labels-disc.nii.gz', dtype = 'uint8') + print(f'\nSaving disc positions in {image_discs.orientation} orientation as {path_template}template_labels-disc.nii.gz\n') # generate template centerline as a npz file param_centerline = ParamCenterline(algo_fitting = algo_fitting, smooth = smooth, degree = degree, minmax = minmax) # centerline params of original template centerline had options that you cannot just provide `get_centerline` with anymroe (algo_fitting = 'nurbs', nurbs_pts_number = 4000, all_slices = False, phys_coordinates = True, remove_outliers = True) _, arr_ctl, arr_ctl_der, _ = get_centerline(image_centerline, param = param_centerline, space = 'phys') centerline_template = Centerline(points_x = arr_ctl[0], points_y = arr_ctl[1], points_z = arr_ctl[2], deriv_x = arr_ctl_der[0], deriv_y = arr_ctl_der[1], deriv_z = arr_ctl_der[2]) - centerline_template.compute_vertebral_distribution(coord_physical) + centerline_template.compute_vertebral_distribution(coord_physical) centerline_template.save_centerline(fname_output = path_template + 'template_label-centerline') print(f'\nSaving template centerline as .npz file (saves all Centerline object information, not just coordinates) as {path_template}template_label-centerline.npz\n') @@ -468,8 +468,8 @@ def straighten_all_subjects(dataset_info, normalized = False): if not os.path.exists(folder_out): os.makedirs(folder_out) fname_image = path_data + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '.nii.gz' - fname_image_seg = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-SC_seg.nii.gz' - fname_image_discs = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-disc.nii.gz' + fname_image_seg = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-SC_mask.nii.gz' + fname_image_discs = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_labels-disc.nii.gz' fname_image_centerline = path_data + 'derivatives/labels/' + subject_name + '/' + dataset_info['data_type'] + '/' + subject_name + dataset_info['suffix_image'] + '_label-centerline.nii.gz' fname_out = subject_name + dataset_info['suffix_image'] + '_straight_norm.nii.gz' if normalized else subject_name + dataset_info['suffix_image'] + '_straight.nii.gz' @@ -484,7 +484,7 @@ def straighten_all_subjects(dataset_info, normalized = False): ' -s ' + fname_input_seg + ' -dest ' + path_template + 'template_label-centerline.nii.gz' + ' -ldisc-input ' + fname_image_discs + - ' -ldisc-dest ' + path_template + 'template_label-disc.nii.gz' + + ' -ldisc-dest ' + path_template + 'template_labels-disc.nii.gz' + ' -ofolder ' + folder_out + ' -o ' + fname_out + ' -disable-straight2curved' + @@ -666,7 +666,7 @@ def main(configuration_file): dataset_info = read_dataset(configuration_file) # generating centerlines - list_centerline = generate_centerline(dataset_info = dataset_info) + list_centerline = generate_centerline(dataset_info = dataset_info) # computing average template centerline and vertebral distribution points_average_centerline, position_template_discs = average_centerline(list_centerline = list_centerline, diff --git a/preprocess_segment.sh b/preprocess_segment.sh index 9558f43..6f4c974 100755 --- a/preprocess_segment.sh +++ b/preprocess_segment.sh @@ -34,8 +34,7 @@ PATH_DATA=$(echo "$json_data" | sed -n 's/.*"path_data": "\(.*\)".*/\1/p') DATA_TYPE=$(echo "$json_data" | sed -n 's/.*"data_type": "\(.*\)".*/\1/p') IMAGE_SUFFIX=$(echo "$json_data" | sed -n 's/.*"suffix_image": "\(.*\)".*/\1/p') CONTRAST=$(echo "$json_data" | sed -n 's/.*"contrast": "\(.*\)".*/\1/p') -PATH_DATASET_OUTPUT="${PATH_DATA}derivatives/labels/${SUBJECT}/${DATA_TYPE}" -mkdir -p ${PATH_DATASET_OUTPUT} +FILE=$PATH_DATA_PROCESSED/$SUBJECT/$DATA_TYPE/${SUBJECT}${IMAGE_SUFFIX}.nii.gz # Uncomment for full verbose # set -v @@ -57,46 +56,48 @@ start=`date +%s` sct_check_dependencies -short # Go to folder where data will be copied and processed -cd $PATH_DATA_PROCESSED +mkdir -p $PATH_DATA_PROCESSED/$SUBJECT/$DATA_TYPE +mkdir -p $PATH_DATA_PROCESSED/derivatives/labels/$SUBJECT/$DATA_TYPE +cd $PATH_DATA_PROCESSED/derivatives/labels/$SUBJECT/$DATA_TYPE # Copy source images -rsync -avzh $PATH_DATA/$SUBJECT/$DATA_TYPE/* . +rsync -avzh $PATH_DATA/$SUBJECT/$DATA_TYPE/${SUBJECT}${IMAGE_SUFFIX}.nii.gz $PATH_DATA_PROCESSED/$SUBJECT/$DATA_TYPE # Segment spinal cord (SC) if does not exist # ====================================================================================================================== -FILE="${SUBJECT}${IMAGE_SUFFIX}.nii.gz" -FILESEG="${SUBJECT}${IMAGE_SUFFIX}_label-SC_seg.nii.gz" +FILESEG="${SUBJECT}${IMAGE_SUFFIX}_label-SC_mask.nii.gz" echo "Looking for segmentation: ${FILESEG}" -if [[ -e "${PATH_DATASET_OUTPUT}/${FILESEG}" ]]; then +if [[ -e "${FILESEG}" ]]; then echo "Found! Using SC segmentation that exists." - sct_qc -i ${FILE} -s "${PATH_DATASET_OUTPUT}/${FILESEG}" -p sct_deepseg_sc -qc ${PATH_QC} -qc-subject ${SUBJECT} + sct_qc -i ${FILE} -s "${FILESEG}" -p sct_deepseg_sc -qc ${PATH_QC} -qc-subject ${SUBJECT} else echo "Not found. Proceeding with automatic segmentation." # Segment spinal cord sct_deepseg_sc -i ${FILE} -o ${FILESEG} -c ${CONTRAST} -qc ${PATH_QC} -qc-subject ${SUBJECT} - mv ${FILESEG} "${PATH_DATASET_OUTPUT}/${FILESEG}" fi + # Label discs if do not exist # ====================================================================================================================== -FILELABEL="${SUBJECT}${IMAGE_SUFFIX}_label-disc.nii.gz" +FILELABEL="${SUBJECT}${IMAGE_SUFFIX}_labels-disc.nii.gz" echo "Looking for disc labels: ${FILELABEL}" -if [[ -e "${PATH_DATASET_OUTPUT}/${FILELABEL}" ]]; then +if [[ -e "${FILELABEL}" ]]; then echo "Found! Using vertebral labels that exist." - sct_qc -i ${FILE} -s "${PATH_DATASET_OUTPUT}/${FILELABEL}" -p sct_label_vertebrae -qc ${PATH_QC} -qc-subject ${SUBJECT} + sct_qc -i ${FILE} -s "${FILELABEL}" -p sct_label_vertebrae -qc ${PATH_QC} -qc-subject ${SUBJECT} else echo "Not found. Proceeding with automatic labeling." # Generate labeled segmentation - sct_label_vertebrae -i ${FILE} -s "${PATH_DATASET_OUTPUT}/${FILESEG}" -ofolder ${PATH_DATASET_OUTPUT} -c ${CONTRAST} -qc "${PATH_QC}" -qc-subject "${SUBJECT}" - mv "${PATH_DATASET_OUTPUT}/${SUBJECT}${IMAGE_SUFFIX}_label-SC_seg_labeled_discs.nii.gz" "${PATH_DATASET_OUTPUT}/${FILELABEL}" - rm "${PATH_DATASET_OUTPUT}/${SUBJECT}${IMAGE_SUFFIX}_label-SC_seg_labeled.nii.gz" + sct_label_vertebrae -i ${FILE} -s "${FILESEG}" -c ${CONTRAST} -qc "${PATH_QC}" -qc-subject "${SUBJECT}" + mv "${SUBJECT}${IMAGE_SUFFIX}_label-SC_mask_labeled_discs.nii.gz" "${FILELABEL}" + rm "${SUBJECT}${IMAGE_SUFFIX}_label-SC_mask_labeled.nii.gz" fi + # Verify presence of output files and write log file if error # ====================================================================================================================== FILES_TO_CHECK=( @@ -104,11 +105,12 @@ FILES_TO_CHECK=( "$FILELABEL" ) for file in "${FILES_TO_CHECK[@]}"; do - if [ ! -e "${PATH_DATASET_OUTPUT}/${file}" ]; then - echo "${PATH_DATASET_OUTPUT}/${file} does not exist" >> "${PATH_LOG}/error.log" + if [ ! -e "${file}" ]; then + echo "${file} does not exist" >> "${PATH_LOG}/error.log" fi done + # Display useful info for the log # ====================================================================================================================== end=`date +%s`