Merge remote-tracking branch 'upstream/main' into nipreps-config

.github/workflows/lint.yml

@@ -32,6 +32,6 @@ jobs:
         run: python -m pip install \
           flake8 flake8-absolute-import flake8-black flake8-docstrings \
           flake8-isort flake8-pyproject flake8-unused-arguments \
-          flake8-use-fstring flake8-warnings pep8-naming
+          flake8-use-fstring pep8-naming
       - name: Check xcp_d
         run: python -m flake8 xcp_d

CITATION.cff

@@ -86,5 +86,5 @@ keywords:
   - BIDS-App
   - Neuroimaging
 license: BSD-3-Clause
-version: 0.4.0
-date-released: '2023-06-12'
+version: 0.5.0
+date-released: '2023-09-13'

Dockerfile

@@ -1,4 +1,4 @@
-FROM pennlinc/xcp_d_build:0.0.6rc15
+FROM pennlinc/xcp_d_build:0.0.7
 
 # Install xcp_d
 COPY . /src/xcp_d

README.rst

@@ -60,7 +60,7 @@ data-releases>`_.
 
 *Please note that XCP is only compatible with HCP-YA versions downloaded c.a. Feb 2023 at the moment.*
 
-.. image:: https://raw.githubusercontent.com/pennlinc/xcp_d/main/docs/_static/schematic_land-01.png
+.. image:: https://raw.githubusercontent.com/pennlinc/xcp_d/main/docs/_static/xcp_paper_fig_1.png
 
 See the `documentation <https://xcp-d.readthedocs.io/en/latest/>`_ for more details.
 

CHANGES.md → docs/changes.md

@@ -1,3 +1,45 @@
+# What's New
+
+## 0.5.0
+
+The 0.5.0 release prepares for the XCP-D manuscript, so I plan to not introduce any backwards-incompatible changes between this release and 1.0.0 (the official paper release).
+<!-- Release notes generated using configuration in .github/release.yml at main -->
+
+### 🛠 Breaking Changes
+* Add CIFTI subcortical atlas to XCP-D by @tsalo in https://github.com/PennLINC/xcp_d/pull/864
+* Change "ciftiSubcortical" atlas name to "HCP" by @tsalo in https://github.com/PennLINC/xcp_d/pull/932
+### 🎉 Exciting New Features
+* Write out top-level sidecar for PennLINC QC file by @tsalo in https://github.com/PennLINC/xcp_d/pull/894
+* Parcellate ReHo, ALFF, and surface morphometric maps by @tsalo in https://github.com/PennLINC/xcp_d/pull/839
+* Refactor dcan/hcp ingestion and add more surface files by @tsalo in https://github.com/PennLINC/xcp_d/pull/887
+* Add "none" option for denoising by @tsalo in https://github.com/PennLINC/xcp_d/pull/879
+* Add `--exact-time` parameter by @tsalo in https://github.com/PennLINC/xcp_d/pull/885
+* Allow white matter surface suffix to be either "_smoothwm" or "_white" by @tsalo in https://github.com/PennLINC/xcp_d/pull/899
+* Add `--fs-license-file` parameter to command line interface by @tsalo in https://github.com/PennLINC/xcp_d/pull/930
+* Generate executive summary figures without `--dcan-qc` by @tsalo in https://github.com/PennLINC/xcp_d/pull/936
+### 🐛 Bug Fixes
+* Standardize executive summary carpet if params is "none" by @tsalo in https://github.com/PennLINC/xcp_d/pull/916
+* Support CIFTI morphometry files and add PNC test data by @tsalo in https://github.com/PennLINC/xcp_d/pull/922
+* Correct ABCD/HCP surface ingression by @tsalo in https://github.com/PennLINC/xcp_d/pull/927
+### Other Changes
+* Add contributing documentation by @tsalo in https://github.com/PennLINC/xcp_d/pull/893
+* Update changelog and CFF file for 0.4.0 by @tsalo in https://github.com/PennLINC/xcp_d/pull/896
+* Change packaging to use hatch-vcs and pyproject.toml by @tsalo in https://github.com/PennLINC/xcp_d/pull/897
+* Fix description of signal denoising method in docs by @tsalo in https://github.com/PennLINC/xcp_d/pull/898
+* Improve QC sidecar contents by @tsalo in https://github.com/PennLINC/xcp_d/pull/900
+* Rename the parcellated ALFF/ReHo outputs by @tsalo in https://github.com/PennLINC/xcp_d/pull/902
+* Remove workaround for nonbinary Nibabies brain masks by @tsalo in https://github.com/PennLINC/xcp_d/pull/905
+* Update Landing Page Figure by @kahinimehta in https://github.com/PennLINC/xcp_d/pull/908
+* Remove unused functions in filemanip module by @tsalo in https://github.com/PennLINC/xcp_d/pull/911
+* Add tests for xcp_d.utils.utils module by @tsalo in https://github.com/PennLINC/xcp_d/pull/910
+* Add tests for xcp_d.utils.execsummary module by @tsalo in https://github.com/PennLINC/xcp_d/pull/912
+* Expand tests by @tsalo in https://github.com/PennLINC/xcp_d/pull/913
+* Test CLI parameter validation by @tsalo in https://github.com/PennLINC/xcp_d/pull/918
+* Refactor collection functions by @tsalo in https://github.com/PennLINC/xcp_d/pull/917
+
+
+**Full Changelog**: https://github.com/PennLINC/xcp_d/compare/04.0...0.5.0
+
 ## 0.4.0
 
 ### 🛠 Breaking Changes
@@ -163,7 +205,8 @@ Two big breaking changes are (1) there is a new `--dcan-qc` flag that determines
 **Full Changelog**: https://github.com/PennLINC/xcp_d/compare/0.2.0...0.3.0
 
 
-# 0.2.2
+## 0.2.2
+
 This is a patch release for the 0.2 series. The main bug being fixed is that using `--dummytime` was causing crashes in the executive summary workflow.
 
 ### 🎉 Exciting New Features
@@ -304,7 +347,7 @@ There is a full list of the changes made between 0.1.3 and 0.2.0 below. However,
 * [TEST] Add tests for outstanding modules by @kahinimehta in https://github.com/PennLINC/xcp_d/pull/529
 * [FIX] Fcon workflow tests are incompatible with changes from main by @kahinimehta in https://github.com/PennLINC/xcp_d/pull/584
 
-## New Contributors
+### New Contributors
 * @tsalo made their first contribution in https://github.com/PennLINC/xcp_d/pull/457
 
 **Full Changelog**: https://github.com/PennLINC/xcp_d/compare/0.1.3...0.2.0

docs/conf.py

@@ -412,6 +412,7 @@
 
 
 def setup(app):
+    """Add extra formatting files."""
     app.add_css_file("theme_overrides.css")
     # We need this for the boilerplate script
     app.add_js_file("https://cdn.rawgit.com/chrisfilo/zenodo.js/v0.1/zenodo.js")

docs/links.rst

@@ -10,3 +10,5 @@
 .. _`Docker installation`: https://docs.docker.com/install/
 .. _`Docker Hub`: https://hub.docker.com/r/pennlinc/xcp_d/tags
 .. _`Singularity`: https://sylabs.io/guides/3.5/user-guide/introduction.html
+.. _`QSIPrep`: https://qsiprep.readthedocs.io
+.. _`ASLPrep`: https://aslprep.readthedocs.io

docs/outputs.rst

@@ -5,16 +5,16 @@
 Outputs of *XCP-D*
 ##################
 
-The  ``xcp_d`` outputs are written out in BIDS format and consist of three main parts.
+The  *XCP-D* outputs are written out in BIDS format and consist of three main parts.
 
 .. admonition:: A note on BIDS compliance
 
-   ``xcp_d`` attempts to follow the BIDS specification as best as possible.
-   However, many ``xcp_d`` derivatives are not currently covered by the specification.
+   *XCP-D* attempts to follow the BIDS specification as best as possible.
+   However, many *XCP-D* derivatives are not currently covered by the specification.
    In those instances, we attempt to follow recommendations from existing BIDS Extension Proposals
    (BEPs), which are in-progress proposals to add new features to BIDS.
 
-   Three BEPs that are of particular use in ``xcp_d`` are
+   Three BEPs that are of particular use in *XCP-D* are
    `BEP012: Functional preprocessing derivatives <https://github.com/bids-standard/bids-specification/pull/519>`_,
    `BEP017: BIDS connectivity matrix data schema <https://docs.google.com/document/d/1ugBdUF6dhElXdj3u9vw0iWjE6f_Bibsro3ah7sRV0GA/edit?usp=sharing>`_,
    and
@@ -24,8 +24,8 @@ The  ``xcp_d`` outputs are written out in BIDS format and consist of three main
    In cases where a derivative type is not covered by an existing BEP,
    we have simply attempted to follow the general principles of BIDS.
 
-   If you discover a problem with the BIDS compliance of ``xcp_d``'s derivatives, please open an
-   issue in the ``xcp_d`` repository.
+   If you discover a problem with the BIDS compliance of *XCP-D*'s derivatives, please open an
+   issue in the *XCP-D* repository.
 
 
 ***************
@@ -44,6 +44,33 @@ The executive summary is based on the DCAN lab's
       sub-<label>[_ses-<label>]_executive_summary.html
 
 
+*************************
+Parcellations and Atlases
+*************************
+
+*XCP-D* produces parcellated anatomical and functional outputs using a series of atlases.
+The individual outputs are documented in the relevant sections of this document,
+with this section describing the atlases themselves.
+
+The atlases currently used in *XCP-D* can be separated into three groups: subcortical, cortical,
+and combined cortical/subcortical.
+The two subcortical atlases are the Tian atlas :footcite:p:`tian2020topographic` and the
+CIFTI subcortical parcellation.
+The cortical atlases are the Glasser :footcite:p:`Glasser_2016` and the
+Gordon :footcite:p:`Gordon_2014`.
+The combined cortical/subcortical atlases are 10 different resolutions of the
+4S (Schaefer Supplemented with Subcortical Structures) atlas.
+
+The 4S atlas combines the Schaefer 2018 cortical atlas (version v0143) :footcite:p:`Schaefer_2017`
+at 10 different resolutions (100, 200, 300, 400, 500, 600, 700, 800, 900, and 1000 parcels) with
+the CIT168 subcortical atlas :footcite:p:`pauli2018high`,
+the Diedrichson cerebellar atlas :footcite:p:`king2019functional`,
+and the HCP thalamic atlas :footcite:p:`najdenovska2018vivo`.
+The 4S atlas is used in the same manner across three PennLINC BIDS Apps:
+XCP-D, QSIPrep_, and ASLPrep_, to produce synchronized outputs across modalities.
+For more information about the 4S atlas, please see https://github.com/PennLINC/AtlasPack.
+
+
 ******************
 Anatomical Outputs
 ******************
@@ -116,7 +143,7 @@ functional connectivity matrices, and resting-state derivatives.
 
 .. important::
 
-   Prior to version 0.4.0, the denoised data outputted by ``xcp_d`` was interpolated,
+   Prior to version 0.4.0, the denoised data outputted by *XCP-D* was interpolated,
    meaning that high-motion volumes were replaced with interpolated data prior to temporal
    filtering.
    **This was a bug.**
@@ -178,6 +205,10 @@ Functional timeseries and connectivity matrices
 
 This includes the atlases used to extract the timeseries.
 
+.. important::
+   Correlation matrices with the ``desc-<INT>volumes`` entity are produced if the ``--exact-time``
+   parameter is used.
+
 .. code-block::
 
    xcp_d/
@@ -193,6 +224,7 @@ This includes the atlases used to extract the timeseries.
             <source_entities>_space-<label>_atlas-<label>_coverage.tsv
             <source_entities>_space-<label>_atlas-<label>_timeseries.tsv
             <source_entities>_space-<label>_atlas-<label>_measure-pearsoncorrelation_conmat.tsv
+            <source_entities>_space-<label>_atlas-<label>_measure-pearsoncorrelation_desc-<INT>volumes_conmat.tsv
 
             # Cifti
             <source_entities>_space-fsLR_atlas-<label>_den-91k_coverage.tsv
@@ -201,6 +233,7 @@ This includes the atlases used to extract the timeseries.
             <source_entities>_space-fsLR_atlas-<label>_den-91k_timeseries.ptseries.nii
             <source_entities>_space-fsLR_atlas-<label>_den-91k_measure-pearsoncorrelation_conmat.tsv
             <source_entities>_space-fsLR_atlas-<label>_den-91k_measure-pearsoncorrelation_conmat.pconn.nii
+            <source_entities>_space-fsLR_atlas-<label>_den-91k_measure-pearsoncorrelation_desc-<INT>volumes_conmat.tsv
 
 
 Resting-state metric derivatives (ReHo and ALFF)
@@ -230,15 +263,15 @@ data.
             <source_entities>_space-<label>_reho.nii.gz
             <source_entities>_space-<label>_alff.nii.gz
             <source_entities>_space-<label>_desc-smooth_alff.nii.gz
-            <source_entities>_space-<label>_atlas-<atlas>_desc-alff_timeseries.tsv
-            <source_entities>_space-<label>_atlas-<atlas>_desc-reho_timeseries.tsv
+            <source_entities>_space-<label>_atlas-<atlas>_alff.tsv
+            <source_entities>_space-<label>_atlas-<atlas>_reho.tsv
 
             # Cifti
             <source_entities>_space-fsLR_den-91k_reho.dscalar.nii
             <source_entities>_space-fsLR_den-91k_alff.dscalar.nii
             <source_entities>_space-fsLR_den-91k_desc-smooth_alff.dscalar.nii
-            <source_entities>_space-fsLR_atlas-<atlas>_desc-alff_timeseries.tsv
-            <source_entities>_space-fsLR_atlas-<atlas>_desc-reho_timeseries.tsv
+            <source_entities>_space-fsLR_atlas-<atlas>_alff.tsv
+            <source_entities>_space-fsLR_atlas-<atlas>_reho.tsv
 
 
 Other outputs include quality control, framewise displacement, and confounds files
@@ -248,6 +281,7 @@ Other outputs include quality control, framewise displacement, and confounds fil
 
    xcp_d/
       desc-linc_qc.json
+
       sub-<label>/[ses-<label>/]
          func/
             # Nifti

docs/usage.rst

@@ -294,21 +294,20 @@ The individual confounds files should be tab-delimited, with one column for each
 and one row for each volume in the data being denoised.
 
 
-Signal Confounds for Non-Aggressive Denoising
-=============================================
+Including Signal Regressors
+===========================
 
 Let's say you have some nuisance regressors that are not necessarily orthogonal to some associated
-regressors that are ostensibly noise.
+regressors that are ostensibly signal.
 For example, if you ran `tedana <https://tedana.readthedocs.io/en/stable/>`_ on multi-echo data,
 you would have a series of "rejected" (noise) and "accepted" (signal) ICA components.
 Because tedana uses a spatial ICA, these components' time series are not necessarily independent,
 and there can be shared variance between them.
 If you want to properly denoise your data using the noise components,
-you need to perform "non-aggressive" denoising so that variance from the signal components is not
-removed as well.
-In non-aggressive denoising, you fit a GLM using both the noise and signal regressors,
-then reconstruct the predicted data using just the noise regressors,
-and finally remove that predicted data from the real data.
+you need to account for the variance they share with the signal components.
+
+XCP-D allows users to include the signal regressors in their custom confounds file,
+so that the noise regressors can be orthogonalized with respect to the signal regressors.
 
 For more information about different types of denoising,
 see `tedana's documentation <https://tedana.readthedocs.io/en/latest/denoising.html>`_,
@@ -317,15 +316,15 @@ and/or `Pruim et al. (2015) <https://doi.org/10.1016/j.neuroimage.2015.02.064>`_
 
 So how do we implement this in XCP-D?
 In order to define regressors that should be treated as signal,
-and thus use non-aggressive denoising instead of the default aggressive denoising,
+and thus orthogonalize the noise regressors with respect to known signals instead of regressing
+them without modification,
 you should include those regressors in your custom confounds file,
 with column names starting with ``signal__`` (lower-case "signal", followed by two underscores).
 
 .. important::
 
-   XCP-D will automatically perform non-aggressive denoising with any nuisance-regressor option
-   that uses AROMA regressors
-   (e.g., ``aroma`` or ``aroma_gsr``).
+   XCP-D will automatically orthogonalize noise regressors with respect to signal regressors
+   with any nuisance-regressor option that uses AROMA regressors (e.g., ``aroma`` or ``aroma_gsr``).
 
 
 Task Regression

docs/workflows.rst

@@ -146,6 +146,8 @@ custom confounds can be added as described in :ref:`usage_custom_confounds`.
 If you want to use custom confounds, without any of the nuisance regressors described here,
 use ``--nuisance-regressors custom``.
 
+If you want to skip the denoising step completely, you can use ``--nuisance-regressors none``.
+
 .. important::
    Starting in version 0.4.0, if motion parameters were filtered earlier in the workflow,
    the filtered motion parameters (including FD, and any squared or derivative regressors)
@@ -225,6 +227,15 @@ use ``--nuisance-regressors custom``.
       - X
       - X
       - X
+   *  - none
+      -
+      -
+      -
+      -
+      -
+      -
+      -
+      -
 
 For more information about confound regressor selection, please refer to :footcite:t:`benchmarkp`.
 
@@ -381,12 +392,8 @@ Parcellation and functional connectivity estimation
 :func:`~xcp_d.workflows.connectivity.init_functional_connectivity_cifti_wf`
 
 The ``filtered, denoised BOLD`` is fed into a functional connectivity workflow,
-which extracts parcel-wise time series from the BOLD using several atlases:
-
-   a.  Schaefer 100,200,300,400,500,600,700,800,900,1000
-   b.  Glasser 360
-   c.  Gordon 333
-   d.  Tian Subcortical Atlas :footcite:p:`tian2020topographic`
+which extracts parcel-wise time series from the BOLD using several atlases.
+These atlases are documented in :doc:`outputs`.
 
 The resulting parcellated time series for each atlas is then used to generate static functional
 connectivity matrices, as measured with Pearson correlation coefficients.
@@ -395,6 +402,30 @@ For CIFTI data, both tab-delimited text file (TSV) and CIFTI versions of the par
 series and correlation matrices are written out.
 
 
+Functional connectivity estimates from specified amounts of data [OPTIONAL]
+---------------------------------------------------------------------------
+
+Functional connectivity estimates may exhibit non-linear relationships with the number of data
+points,
+such that including a regressor controlling for the number of post-censoring volumes per run in
+group-level models may not adequately address the issue.
+
+In :footcite:t:`eggebrecht2017joint` and :footcite:t:`feczko2021adolescent`,
+the authors' solution was to randomly select a subset of volumes from each run before calculating
+correlations, so that every run had the same number of data points contributing to its functional
+connectivity estimate.
+
+We have implemented this behavior via the optional ``--exact-time`` parameter, which allows the
+user to provide a list of durations, in seconds, to be used for functional connectivity estimates.
+These subsampled correlation matrices will be written out with ``desc-<numberOfVolumes>volumes``
+in the filenames.
+The correlation matrices *without* the ``desc`` entity still include all of the post-censoring
+volumes.
+
+The ``--random-seed`` parameter can control the random seed used to select the reduced set of \
+volumes, which improves reproducibility.
+
+
 Smoothing [OPTIONAL]
 ====================
 :func:`~xcp_d.workflows.postprocessing.init_resd_smoothing_wf`

pyproject.toml

@@ -74,7 +74,6 @@ tests = [
     "flake8-pyproject",
     "flake8-unused-arguments",
     "flake8-use-fstring",
-    "flake8-warnings",
     "pep8-naming",
     "pytest",
     "pytest-cov",
@@ -166,11 +165,13 @@ per-file-ignores = [
 ]
 
 [tool.pytest.ini_options]
-addopts = '-m "not ds001419_nifti and not ds001419_cifti and not ds001419_cifti_t2wonly and not fmriprep_without_freesurfer and not nibabies"'
+addopts = '-m "not ds001419_nifti and not ds001419_cifti and not pnc_nifti and not pnc_cifti and not pnc_cifti_t2wonly and not fmriprep_without_freesurfer and not nibabies"'
 markers = [
-    "ds001419_nifti: mark integration test for fMRIPrep derivatives with NIFTI settings",
-    "ds001419_cifti: mark integration test for fMRIPrep derivatives with CIFTI settings",
-    "ds001419_cifti_t2wonly: mark integration test for fMRIPrep derivatives with CIFTI settings and a simulated T2w file",
+    "ds001419_nifti: mark NIfTI integration test for fMRIPrep derivatives from ds001419",
+    "ds001419_cifti: mark CIFTI integration test for fMRIPrep derivatives from ds001419",
+    "pnc_nifti: mark integration test for fMRIPrep derivatives with NIFTI settings",
+    "pnc_cifti: mark integration test for fMRIPrep derivatives with CIFTI settings",
+    "pnc_cifti_t2wonly: mark integration test for fMRIPrep derivatives with CIFTI settings and a simulated T2w file",
     "fmriprep_without_freesurfer: mark integration test for fMRIPrep derivatives without FreeSurfer",
     "nibabies: mark integration test for nibabies derivatives",
 ]