Merge branch 'Inria-Empenn:main' into main

INSTALL.md

@@ -40,18 +40,18 @@ datalad get data/original/ds001734/derivatives/fmriprep/sub-00[1-4] -J 12
 
 ## 4 - Set up the environment
 
-[Install Docker](https://docs.docker.com/engine/install/) then pull the Docker image :
+[Install Docker](https://docs.docker.com/engine/install/) then pull the nipype Docker image :
 
 ```bash
-docker pull elodiegermani/open_pipeline:latest
+docker pull nipype/nipype
 ```
 
 Once it's done you can check the image is available on your system :
 
 ```bash
 docker images
-   REPOSITORY                               TAG       IMAGE ID        CREATED         SIZE
-   docker.io/elodiegermani/open_pipeline    latest    0f3c74d28406    9 months ago    22.7 GB
+   REPOSITORY                 TAG       IMAGE ID        CREATED         SIZE
+   docker.io/nipype/nipype    latest    0f3c74d28406    9 months ago    22.7 GB
 ```
 
 > [!NOTE]  
@@ -63,7 +63,7 @@ Start a Docker container from the Docker image :
 
 ```bash
 # Replace PATH_TO_THE_REPOSITORY in the following command (e.g.: with /home/user/dev/narps_open_pipelines/)
-docker run -it -v PATH_TO_THE_REPOSITORY:/home/neuro/code/ elodiegermani/open_pipeline
+docker run -it -v PATH_TO_THE_REPOSITORY:/home/neuro/code/ nipype/nipype
 ```
 
 Install NARPS Open Pipelines inside the container :
@@ -74,12 +74,36 @@ cd /home/neuro/code/
 pip install .
 ```
 
-Finally, you are able to run pipelines :
+Finally, you are able to use the scripts of the project :
+
+* `narps_open_runner`: run pipelines
+* `narps_open_tester`: run a pipeline and test its results against original ones from the team
+* `narps_description`: get the textual description made by a team
+* `narps_results`: download the original results from teams
+* `narps_open_status`: get status information about the development process of the pipelines
 
 ```bash
-python narps_open/runner.py
-  usage: runner.py [-h] -t TEAM (-r RSUBJECTS | -s SUBJECTS [SUBJECTS ...] | -n NSUBJECTS) [-g | -f] [-c]
+# Run the pipeline for team 2T6S, with 40 subjects
+narps_open_runner -t 2T6S -n 40
+
+# Run the pipeline for team 08MQ, compare results with original ones,
+#   and produces a report with correlation values.
+narps_open_tester -t 08MQ
+
+# Get the description of team C88N in markdown formatting
+narps_description -t C88N --md
+
+# Download the results from all teams
+narps_results -a
+
+#  Get the pipeline work status information in json formatting
+narps_open_status --json
 ```
 
 > [!NOTE]  
-> For further information, read this documentation page [docs/running.md](docs/running.md).
+> For further information about these command line tools, read the corresponding documentation pages.
+> * `narps_open_runner` : [docs/running.md](docs/running.md)
+> * `narps_open_tester` : [docs/testing.md](docs/testing.md#command-line-tool)
+> * `narps_description` : [docs/description.md](docs/description.md)
+> * `narps_results` : [docs/data.md](docs/data.md#results-from-narps-teams)
+> * `narps_open_status` : [docs/status.md](docs/status.md)

README.md

@@ -17,9 +17,11 @@
 
 **The goal of the NARPS Open Pipelines project is to create a codebase reproducing the 70 pipelines of the NARPS study (Botvinik-Nezer et al., 2020) and share this as an open resource for the community**. 
 
-We base our reproductions on the [original descriptions provided by the teams](https://github.com/poldrack/narps/blob/1.0.1/ImageAnalyses/metadata_files/analysis_pipelines_for_analysis.xlsx) and test the quality of the reproductions by comparing our results with the original results published on NeuroVault.
+We base our reproductions on the original descriptions provided by the teams and test the quality of the reproductions by comparing our results with the original results published on NeuroVault.
 
-:vertical_traffic_light: See [the pipeline dashboard](https://github.com/Inria-Empenn/narps_open_pipelines/wiki/pipeline_status) to view our current progress at a glance.
+Find more information about the NARPS study [here](docs/narps.md).
+
+:vertical_traffic_light: See [the pipeline dashboard](https://github.com/Inria-Empenn/narps_open_pipelines/wiki/pipeline_status) to view our current progress at a glance !
 
 ## Contributing
 

docs/data.md

@@ -67,28 +67,31 @@ for team in teams:
     collection.rectify() # Rectified versions are created
 ```
 
+> [!TIP]
+> In the following examples, use `narps_results` or `python narps_open/data/results` indifferently to launch the command line tool.
+
 ```bash
 # From the command line
-$ python narps_open/data/results -h
-usage: results [-h] (-t TEAMS [TEAMS ...] | -a) [-r]
+narps_results -h
+    usage: results [-h] (-t TEAMS [TEAMS ...] | -a) [-r]
 
-Get Neurovault collection of results from NARPS teams.
+    Get Neurovault collection of results from NARPS teams.
 
-options:
-  -h, --help            show this help message and exit
-  -t TEAMS [TEAMS ...], --teams TEAMS [TEAMS ...]
-                        a list of team IDs
-  -a, --all             download results from all teams
-  -r, --rectify         rectify the results
+    options:
+      -h, --help            show this help message and exit
+      -t TEAMS [TEAMS ...], --teams TEAMS [TEAMS ...]
+                            a list of team IDs
+      -a, --all             download results from all teams
+      -r, --rectify         rectify the results
 
 # Either download all collections
-python narps_open/utils/results -a
+narps_results -a
 
 # Or select the ones you need
-python narps_open/utils/results -t 2T6S C88N L1A8
+narps_results -t 2T6S C88N L1A8
 
 # Download and rectify the collections
-python narps_open/utils/results -r -t 2T6S C88N L1A8
+narps_results -r -t 2T6S C88N L1A8
 ```
 
 The collections are also available [here](https://zenodo.org/record/3528329/) as one release on Zenodo that you can download.

docs/description.md

@@ -12,8 +12,11 @@ The class `TeamDescription` of module `narps_open.data.description` acts as a pa
 
 You can use the command-line tool as so. Option `-t` is for the team id, option `-d` allows to print only one of the sub parts of the description among : `general`, `exclusions`, `preprocessing`, `analysis`, `categorized_for_analysis`, `derived`, and `comments`. Options `--json` and `--md` allow to choose the export format you prefer between JSON and Markdown.
 
+> [!TIP]
+> In the following examples, use `narps_description` or `python narps_open/data/description` indifferently to launch the command line tool.
+
 ```bash
-python narps_open/data/description -h
+narps_description -h
 # usage: __init__.py [-h] -t TEAM [-d {general,exclusions,preprocessing,analysis,categorized_for_analysis,derived,comments}]
 #
 # Get description of a NARPS pipeline.
@@ -26,7 +29,7 @@ python narps_open/data/description -h
 #  --json                output team description as JSON
 #  --md                  output team description as Markdown
 
-python narps_open/data/description -t 2T6S --json
+narps_description -t 2T6S --json
 # {
 #     "general.teamID": "2T6S",
 #     "general.NV_collection_link": "https://neurovault.org/collections/4881/",
@@ -41,7 +44,7 @@ python narps_open/data/description -t 2T6S --json
 #     "preprocessing.preprocessing_order": "We used the provided preprocessed data by fMRIPprep 1.1.4 (Esteban, Markiewicz, et al. (2018); Esteban, Blair, et al. (2018); RRID:SCR_016216), which is based on Nipype 1.1.1 (Gorgolewski et al. (2011); Gorgolewski et al. (2018); RRID:SCR_002502) and we additionally conducted a spatial smoothing using the provided preprocessed data set and SPM12. Here, we attach the preprocessing steps described in the provided data set. \nAnatomical data preprocessing\nThe T1-weighted (T1w) image was corrected for intensity non-uniformity (INU) using N4BiasFieldCorrection (Tustison et al. 2010, ANTs 2.2.0), and used as T1w-reference throughout the workflow. The T1w-reference was then skull-stripped using antsBrainExtraction.sh (ANTs 2.2.0), using OASIS as target template. Brain surfaces we
 # ...
 
-python narps_open/data/description -t 2T6S -d general --json
+narps_description -t 2T6S -d general --json
 # {
 #    "teamID": "2T6S",
 #    "NV_collection_link": "https://neurovault.org/collections/4881/",
@@ -53,7 +56,7 @@ python narps_open/data/description -t 2T6S -d general --json
 #    "general_comments": "NA"
 # }
 
-python narps_open/data/description -t 2T6S --md
+narps_description -t 2T6S --md
 # # NARPS team description : 2T6S
 # ## General
 # * `teamID` : 2T6S

docs/environment.md

@@ -2,12 +2,12 @@
 
 ## The Docker container :whale:
 
-The NARPS Open Pipelines project is build upon several dependencies, such as [Nipype](https://nipype.readthedocs.io/en/latest/) but also the original software packages used by the pipelines (SPM, FSL, AFNI...). Therefore, we created a Docker container based on [Neurodocker](https://github.com/ReproNim/neurodocker) that contains software dependencies.
+The NARPS Open Pipelines project is build upon several dependencies, such as [Nipype](https://nipype.readthedocs.io/en/latest/) but also the original software packages used by the pipelines (SPM, FSL, AFNI...). Therefore we recommend to use the [`nipype/nipype` Docker image](https://hub.docker.com/r/nipype/nipype/) that contains all the required software dependencies.
 
-The simplest way to start the container using the command below :
+The simplest way to start the container is by using the command below :
 
 ```bash
-docker run -it elodiegermani/open_pipeline
+docker run -it nipype/nipype
 ```
 
 From this command line, you need to add volumes to be able to link with your local files (code repository).
@@ -16,7 +16,7 @@ From this command line, you need to add volumes to be able to link with your loc
 # Replace PATH_TO_THE_REPOSITORY in the following command (e.g.: with /home/user/dev/narps_open_pipelines/)
 docker run -it \
            -v PATH_TO_THE_REPOSITORY:/home/neuro/code/ \
-           elodiegermani/open_pipeline
+           nipype/nipype
 ``` 
 
 ## Use Jupyter with the container
@@ -27,7 +27,7 @@ If you wish to use [Jupyter](https://jupyter.org/) to run the code, a port forwa
 docker run -it \
            -v PATH_TO_THE_REPOSITORY:/home/neuro/code/ \
            -p 8888:8888 \
-           elodiegermani/open_pipeline
+           nipype/nipype
 ``` 
 
 Then, from inside the container :

docs/narps.md

@@ -0,0 +1,93 @@
+# More information about the NARPS study
+
+This page aims at summarizing the NARPS study [(Botvinik-Nezer et al., 2020)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7771346/) for the future contributors of NARPS Open Pipelines.
+
+In the following, the citations come from the associated data article : [Botvinik-Nezer, R. et al. (2019), 'fMRI data of mixed gambles from the Neuroimaging Analysis Replication and Prediction Study', Scientific Data](https://www.nature.com/articles/s41597-019-0113-7).
+
+## Context
+
+> In the Neuroimaging Analysis Replication and Prediction Study (NARPS), we aim to provide the first scientific evidence on the variability of results across analysis teams in neuroscience.
+
+From this starting point, 70 teams were asked to analyse the same dataset, providing their methods and results to be later analysed and compared. Nine hypotheses where to be tested and a binary decision for each one had to be reported, whether it was significantly supported based on a whole-brain analysis.
+
+## The data
+
+NARPS is based on a dataset of task-fMRI with 108 participants, 4 runs for each.
+
+> For each participant, the dataset includes an anatomical (T1 weighted) scan and fMRI as well as behavioral data from four runs of the task. The dataset is shared through OpenNeuro and is formatted according to the Brain Imaging Data Structure (BIDS) standard. Data pre-processed with fMRIprep and quality control reports are also publicly shared.
+
+### Raw functional volumes
+
+> Each run consists of 64 trials performed during an fMRI scanning run lasting 453 seconds and comprising 453 volumes (given the repetition time of one second).
+
+For each participant, the associated data (4D volumes) is :
+* `data/original/ds001734/sub-*/func/sub-*_task-MGT_run-01_bold.nii.gz`
+* `data/original/ds001734/sub-*/func/sub-*_task-MGT_run-02_bold.nii.gz`
+* `data/original/ds001734/sub-*/func/sub-*_task-MGT_run-03_bold.nii.gz`
+* `data/original/ds001734/sub-*/func/sub-*_task-MGT_run-04_bold.nii.gz`
+
+### Event data
+
+> On each trial, participants were presented with a mixed gamble entailing an equal 50% chance of gaining one amount of money or losing another amount.
+
+For each participant, the associated data (events files) is :
+* `data/original/ds001734/sub-*/func/sub-*_task-MGT_run-01_events.tsv`
+* `data/original/ds001734/sub-*/func/sub-*_task-MGT_run-02_events.tsv`
+* `data/original/ds001734/sub-*/func/sub-*_task-MGT_run-03_events.tsv`
+* `data/original/ds001734/sub-*/func/sub-*_task-MGT_run-04_events.tsv`
+
+This file contains the onsets, response time, and response of the participant, as well as the amount of money proposed (gain and loss) for each trial.
+
+### Preprocessed data
+
+> The pre-processed data included in this dataset were preprocessed using fMRIprep version 1.1.4, which is based on Nipype 1.1.1
+
+For each participant, the associated data (preprocessed volumes, confounds, brain masks, ...) is under :
+* `data/original/ds001734/derivatives/fmriprep/sub-*/func/`
+
+Some teams chose to use this pre-processed data directly as inputs for the statistical analysis, while other performed their own method of pre-processing.
+
+### Task-related data
+
+The associated data (task and dataset description) is :
+* `data/original/ds001734/T1w.json`
+* `data/original/ds001734/task-MGT_bold.json`
+* `data/original/ds001734/task-MGT_sbref.json`
+* `data/original/ds001734/dataset_description.json`
+
+> [!TIP]
+> The `narps_open.data.task` module helps parsing this data.
+
+Furthermore, the participants were assigned to a condition, either *equalRange* or *equalIndifference* :
+
+> Possible gains ranged between 10 and 40 ILS (in increments of 2 ILS) in the equal indifference condition or 5–20 ILS (in increments of 1 ILS) in the equal range condition, while possible losses ranged from 5–20 ILS (in increments of 1 ILS) in both conditions.
+
+The repartition is stored in :
+* `data/original/ds001734/participants.tsv`
+
+> [!TIP]
+> The `narps_open.data.participants` module helps parsing this data.
+
+## The outputs
+
+Each of the team participating in NARPS had to provide a COBIDS-compliant *textual description* of its analysis of the dataset, as well as the results from it.
+
+All the descriptions are gathered in the [analysis_pipelines_for_analysis.xlsx](https://github.com/poldrack/narps/blob/1.0.1/ImageAnalyses/metadata_files/analysis_pipelines_for_analysis.xlsx) in the NARPS repository on GitHub.
+
+> [!TIP]
+> We developed a tool to easily parse this file, see [docs/description.md](docs/description.md) for more details on how to use it.
+
+Results data submitted by all the teams is available [on Zenodo](https://zenodo.org/records/3528329#.Y7_H1bTMKBT)
+
+> [!TIP]
+> This data is included in the NARPS Open Pipelines repository under [data/results](data/results), and we developed a tool to access it easily : see the dedicated section in [docs/data.md](docs/data.md#results-from-narps-teams) for more details.
+
+## Useful resources
+
+* The website dedicated to the study - [www.narps.info](https://www.narps.info/)
+* The article - [Botvinik-Nezer, R. et al. (2020), 'Variability in the analysis of a single neuroimaging dataset by many teams', Nature](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7771346/)
+* The associated data article - [Botvinik-Nezer, R. et al. (2019), 'fMRI data of mixed gambles from the Neuroimaging Analysis Replication and Prediction Study', Scientific Data](https://www.nature.com/articles/s41597-019-0113-7)
+* The GitHub page for the NARPS repository. This code was used to generate the published results - [github.com/poldrack/narps](https://github.com/poldrack/narps)
+* The snapshot of this code base [on Zenodo](https://zenodo.org/records/3709273#.Y2jVkCPMIz4)
+* The dataset [on OpenNeuro](https://openneuro.org/datasets/ds001734/versions/1.0.5)
+* Results data submitted by all the teams [on Zenodo](https://zenodo.org/records/3528329#.Y7_H1bTMKBT)

docs/pipelines.md

@@ -149,4 +149,13 @@ First have a look at the [testing page of the documentation](/docs/testing.md).
 All tests must be contained in a single file named `tests/pipelines/test_team_<team_id>.py`. You can start by copy-pasting the template file : [tests/pipelines/templates/template_test.py](/tests/pipelines/templates/template_test.py) inside the `tests/pipelines/` directory, renaming it accordingly. Then, follow the instructions and tips inside the template and don't forget to replace `XXXX` with the actual team id, inside the document.
 
 > [!NOTE]  
-> Feel free to have a look at [tests/pipelines/test_team_2T6S.py](/tests/pipelines/test_team_2T6S.py), which is the file containing all the automatic tests for the 2T6S pipeline : it gives an example.
+> Feel free to have a look at [tests/pipelines/test_team_C88N.py](/tests/pipelines/test_team_C88N.py), which is the file containing all the automatic tests for the C88N pipeline : it gives an example.
+
+Your test file must contain the following methods:
+* one testing the execution of the pipeline and comparing the results with the original ones. This can be simply achieved with this line :
+```python
+helpers.test_pipeline_evaluation('C88N')
+```
+* one for each method you declared in the pipeline class (e.g.: testing the `get_subject_information` method)
+* one testing the instantiation of your pipeline (see `test_create` in `test_team_C88N.py`)
+* one testing the `get_*_outputs` methods of your pipeline (see `test_outputs` in `test_team_C88N.py`)

docs/running.md

@@ -2,10 +2,13 @@
 
 ## Using the runner application
 
-The `narps_open.runner` module allows to run pipelines from the command line :
+The `narps_open.runner` module allows to run pipelines from the command line.
+
+> [!TIP]
+> In the following examples, use `narps_open_runner` or `python narps_open/runner.py` indifferently to launch the command line tool.
 
 ```bash
-python narps_open/runner.py -h
+narps_open_runner -h
 	usage: runner.py [-h] -t TEAM (-r RANDOM | -s SUBJECTS [SUBJECTS ...]) [-g | -f]
 
 	Run the pipelines from NARPS.
@@ -19,13 +22,14 @@ python narps_open/runner.py -h
 	  -f, --first           run the first levels only (preprocessing + subjects + runs)
 	  -c, --check           check pipeline outputs (runner is not launched)
 
-python narps_open/runner.py -t 2T6S -s 001 006 020 100
-python narps_open/runner.py -t 2T6S -r 4
-python narps_open/runner.py -t 2T6S -r 4 -f
-python narps_open/runner.py -t 2T6S -r 4 -f -c # Check the output files without launching the runner
+narps_open_runner -t 2T6S -s 001 006 020 100
+narps_open_runner -t 2T6S -r 4
+narps_open_runner -t 2T6S -r 4 -f
+narps_open_runner -t 2T6S -r 4 -f -c # Check the output files without launching the runner
 ```
 
-In this usecase, the paths where to store the outputs and to the dataset are picked by the runner from the [configuration](docs/configuration.md).
+> [!NOTE]
+> In this usecase, the paths where to store the outputs and to the dataset are picked by the runner from the [configuration](docs/configuration.md).
 
 ## Using the `PipelineRunner` object
 

docs/status.md

@@ -46,8 +46,11 @@ report.markdown() # Returns a string containing the markdown
 
 You can also use the command-line tool as so.
 
+> [!TIP]
+> In the following examples, use `narps_open_status` or `python narps_open/utils/status.py` indifferently to launch the command line tool.
+
 ```bash
-python narps_open/utils/status -h
+narps_open_status -h
 # usage: status.py [-h] [--json | --md]
 # 
 # Get a work progress status report for pipelines.
@@ -57,7 +60,7 @@ python narps_open/utils/status -h
 #   --json      output the report as JSON
 #   --md        output the report as Markdown
 
-python narps_open/utils/status --json
+narps_open_status --json
 # {
 #     "08MQ": {
 #         "softwares": "FSL",
@@ -83,7 +86,7 @@ python narps_open/utils/status --json
 #     },
 # ...
 
-python narps_open/utils/status --md
+narps_open_status --md
 # ...
 # | team_id | status | main software | fmriprep used ? | related issues | related pull requests | excluded from NARPS analysis | reproducibility |
 # | --- |:---:| --- | --- | --- | --- | --- | --- |