Convert pipeline configs from JSON to YAML (#84)

* Switch to YAMLs for the pipelines. * Replace the existing pipelines with YAMLs. * Update docs. * Update base.py * Match the example YAMLs to old ones.
PennLINC · Aug 27, 2024 · e530fe8 · e530fe8
1 parent 608c368
commit e530fe8
Show file tree

Hide file tree

Showing 46 changed files with 1,361 additions and 1,628 deletions.
diff --git a/docs/building_workflows.rst b/docs/building_workflows.rst
@@ -6,8 +6,8 @@
 Custom Reconstruction Workflows
 ###############################
 
-QSIRecon workflows are defined in ``json`` files. The :ref:`builtin_workflows`
-can be found in the QSIRecon ``json`` format on
+QSIRecon workflows are defined in ``YAML`` files. The :ref:`builtin_workflows`
+can be found in the QSIRecon ``YAML`` format on
 `github <https://github.com/PennLINC/qsirecon/tree/main/qsirecon/data/pipelines>`_,
 and are a great place to get started with customizing parts of these workflows.
 
@@ -19,15 +19,19 @@ Pipeline-level metadata
 
 At the root level of the Recon Spec there are
 
-.. code-block:: json
-
-  {
-    "name": "dsistudio_pipeline",
-    "anatomical": ["mrtrix_5tt_hsvs"],
-    "atlases": ["schaefer100" "schaefer200", "schaefer400", "brainnetome246",
-                "aicha384", "gordon333", "aal116"],
-    "nodes": []
-  }
+.. code-block:: yaml
+  name: dsistudio_pipeline
+  anatomical:
+  - mrtrix_5tt_hsvs
+  atlases:
+  - schaefer100
+  - schaefer200
+  - schaefer400
+  - brainnetome246
+  - aicha384
+  - gordon333
+  - aal116
+  nodes: []
 
 The ``"name"`` element defines the name of the pipeline. There will be a directory in your working
 directory with this name. The ``"anatomical"`` field lists additional things to compute
@@ -54,24 +58,23 @@ For example, if you want to perform CSD there are other steps that should
 happen before actually fitting spherical harmonics to the data.
 The entry for this in the ``"nodes"`` list could look like:
 
-.. code-block:: json
-
-  {
-    "name": "msmt_csd",
-    "software": "MRTrix3",
-    "action": "csd",
-    "qsirecon_suffix": "MRtrix3_act-FAST",
-    "parameters": {
-      "mtnormalize": true,
-      "response": {
-        "algorithm": "dhollander"
-      },
-      "fod": {
-        "algorithm": "msmt_csd",
-        "max_sh": [8, 8, 8]
-      }
-    }
-  },
+.. code-block:: yaml
+
+  nodes:
+  -   action: csd
+      name: msmt_csd
+      parameters:
+          fod:
+              algorithm: msmt_csd
+              max_sh:
+              - 8
+              - 8
+              - 8
+          mtnormalize: true
+          response:
+              algorithm: dhollander
+      qsirecon_suffix: MRtrix3_act-FAST
+      software: MRTrix3
 
 QSIRecon figures out which software to use based on the values specified
 for ``"software"`` and ``"action"``. The full list of recognized actions
@@ -99,21 +102,19 @@ another node, you can pass them by matching the ``"input"`` field to the
 name of the upstream node. Here is an example connecting a CSD calculation
 to MRtrix3 tractography.
 
-.. code-block:: json
-
-  {
-    "name": "msmt_csd",
-    "software": "MRTrix3",
-    "action": "csd",
-    "qsirecon_suffix": "MRtrix3_act-FAST"
-  },
-    {
-      "name": "track_ifod2",
-      "software": "MRTrix3",
-      "action": "tractography",
-      "qsirecon_suffix": "MRtrix3_act-FAST",
-      "input": "msmt_csd"
-    },
+.. code-block:: yaml
+
+  nodes:
+  -   action: csd
+      name: msmt_csd
+      qsirecon_suffix: MRtrix3_act-FAST
+      software: MRTrix3
+
+  -   action: tractography
+      input: msmt_csd
+      name: track_ifod2
+      qsirecon_suffix: MRtrix3_act-FAST
+      software: MRTrix3
 
 .. note::
     There can only be zero (inputs come from the input data) or one
@@ -132,35 +133,26 @@ spaces with a "template mapper". Suppose I wanted to fit a NODDI model and
 a DKI model. Then I wanted to transform their model-derived parameters to
 the template space used in QSIPrep. My ``"nodes"`` might look like:
 
-.. code-block:: json
-
-  "nodes": [
-    {
-      "name": "dipy_dki",
-      "software": "Dipy",
-      "action": "DKI_reconstruction",
-      "qsirecon_suffix": "DIPYDKI"
-    },
-    {
-      "name": "fit_noddi",
-      "action": "fit_noddi",
-      "software": "AMICO",
-      "qsirecon_suffix": "NODDI"
-    },
-    {
-      "name": "template_map",
-      "software": "qsirecon",
-      "action": "template_map",
-      "input": "qsirecon",
-      "scalars_from": [
-        "dipy_dki",
-        "fit_noddi"
-      ],
-      "parameters": {
-        "interpolation": "NearestNeighbor"
-      }
-    }
-  ]
+.. code-block:: yaml
+
+  nodes:
+  -   action: DKI_reconstruction
+      name: dipy_dki
+      qsirecon_suffix: DIPYDKI
+      software: Dipy
+  -   action: fit_noddi
+      name: fit_noddi
+      qsirecon_suffix: NODDI
+      software: AMICO
+  -   action: template_map
+      input: qsirecon
+      name: template_map
+      parameters:
+          interpolation: NearestNeighbor
+      scalars_from:
+      - dipy_dki
+      - gqi_scalars
+      software: qsirecon
 
 By listing the names of the scalar-producing nodes in the ``"scalars_from"`` field
 you will end up with the scalars in both subject native and template space in the
@@ -182,39 +174,29 @@ for bundle mapping are
 
 Here is a small example where we use autotrack bundles:
 
-.. code-block:: json
-
-  "nodes": [
-    {
-      "name": "dipy_dki",
-      "software": "Dipy",
-      "action": "DKI_reconstruction",
-      "qsirecon_suffix": "DIPYDKI"
-    },
-    {
-      "name": "fit_noddi",
-      "action": "fit_noddi",
-      "software": "AMICO",
-      "qsirecon_suffix": "NODDI"
-    },
-    {
-      "name": "autotrackgqi",
-      "software": "DSI Studio",
-      "action": "autotrack",
-      "input": "dsistudio_gqi",
-      "qsirecon_suffix": "DSIStudio"
-    },
-    {
-      "name": "bundle_means",
-      "software": "qsirecon",
-      "action": "bundle_map",
-      "input": "autotrackgqi",
-      "scalars_from": [
-        "dipy_dki",
-        "fit_noddi"
-      ]
-    }
-  ]
+.. code-block:: yaml
+
+  nodes:
+  -   action: DKI_reconstruction
+      name: dipy_dki
+      qsirecon_suffix: DIPYDKI
+      software: Dipy
+  -   action: fit_noddi
+      name: fit_noddi
+      qsirecon_suffix: NODDI
+      software: AMICO
+  -   action: autotrack
+      input: dsistudio_gqi
+      name: autotrackgqi
+      qsirecon_suffix: DSIStudio
+      software: DSI Studio
+  -   action: bundle_map
+      input: autotrackgqi
+      name: bundle_means
+      scalars_from:
+      - dipy_dki
+      - fit_noddi
+      software: qsirecon
 
 This will produce a tsv with the NODDI and DKI scalars summarized for
 each bundle produced by autotrack.

diff --git a/docs/input_data.rst b/docs/input_data.rst
@@ -30,8 +30,8 @@ skull-stripped ``brain.mgz`` is rigidly registered to ``dwiref`` of each
 preprocessed DWI. The FreeSurfer brain mask is resampled to the grid of the DWI.
 
 If structural connectivity is calculated during the reconstruction workflow
-(or any atlases are specified in the ``"anatomical": []`` section of the
-workflow's ``.json`` file), the coregistered-to-DWI ``brain.mgz`` image will be
+(or any atlases are specified in the ``anatomical:`` section of the
+workflow's ``.yaml`` file), the coregistered-to-DWI ``brain.mgz`` image will be
 normalized to the MNI152NLin2009cAsym template using ``antsRegistration``.
 The reverse transform is used to get parcellations aligned to the DWI.
 

diff --git a/docs/update_scalardefs.py b/docs/update_scalardefs.py
@@ -18,21 +18,21 @@ def scalars_to_csv(scalar_def, output_csv):
     df.to_csv(output_csv, header=False, index=None)
 
 
-## TORTOISE.json
+## TORTOISE.yaml
 scalars_to_csv(recon_scalars.tortoise_scalars, "recon_scalars/tortoise.csv")
-## amico_noddi.json
+## amico_noddi.yaml
 scalars_to_csv(recon_scalars.amico_scalars, "recon_scalars/amico_noddi.csv")
-## csdsi_3dshore.json
+## csdsi_3dshore.yaml
 scalars_to_csv(recon_scalars.brainsuite_3dshore_scalars, "recon_scalars/csdsi_3dshore.csv")
-## dipy_3dshore.json
+## dipy_3dshore.yaml
 scalars_to_csv(recon_scalars.brainsuite_3dshore_scalars, "recon_scalars/dipy_3dshore.csv")
-## dipy_dki.json
+## dipy_dki.yaml
 scalars_to_csv(recon_scalars.dipy_dki_scalars, "recon_scalars/dipy_dki.csv")
-## dipy_mapmri.json
+## dipy_mapmri.yaml
 scalars_to_csv(recon_scalars.dipy_mapmri_scalars, "recon_scalars/dipy_mapmri.csv")
-## dsi_studio_gqi.json
+## dsi_studio_gqi.yaml
 scalars_to_csv(recon_scalars.dsistudio_scalars, "recon_scalars/dsi_studio_gqi.csv")
-## hbcd_scalar_maps.json
+## hbcd_scalar_maps.yaml
 scalars_to_csv(
     recon_scalars.amico_scalars
     | recon_scalars.dipy_dki_scalars

diff --git a/qsirecon/data/pipelines/TORTOISE.json b/qsirecon/data/pipelines/TORTOISE.json
diff --git a/qsirecon/data/pipelines/TORTOISE.yaml b/qsirecon/data/pipelines/TORTOISE.yaml
@@ -0,0 +1,19 @@
+anatomical: []
+atlases: []
+name: tortoise_recon
+nodes:
+-   action: estimate
+    input: qsirecon
+    name: tortoise_dtmapmri
+    parameters:
+        big_delta: null
+        estimate_mapmri:
+            map_order: 4
+        estimate_tensor:
+            bval_cutoff: 2400
+            write_cs: true
+        estimate_tensor_separately: true
+        small_delta: null
+    qsirecon_suffix: TORTOISE
+    software: TORTOISE
+space: T1w
diff --git a/qsirecon/data/pipelines/amico_noddi.json b/qsirecon/data/pipelines/amico_noddi.json
diff --git a/qsirecon/data/pipelines/amico_noddi.yaml b/qsirecon/data/pipelines/amico_noddi.yaml
@@ -0,0 +1,14 @@
+atlases: []
+description: Runs the AMICO implementation of NODDI
+name: amico_noddi
+nodes:
+-   action: fit_noddi
+    input: qsirecon
+    name: fit_noddi
+    parameters:
+        dIso: 0.003
+        dPar: 0.0017
+        isExvivo: false
+    qsirecon_suffix: NODDI
+    software: AMICO
+space: T1w