Merge branch 'master' into conp-bot/MRI_data_for__Stress_inducible_ph…

…osphoprotein_1__HOP_STI1_STIP1__regulates_the_spreading__aggregation__and_toxicity_of_α_synuclein_in_vivo_

.github/workflows/docker_build.yml

@@ -14,15 +14,15 @@ jobs:
 
     steps:
       - name: Setup Docker Buildx
-        uses: docker/setup-buildx-action@v2
+        uses: docker/setup-buildx-action@v3
       - name: Login to DockerHub
-        uses: docker/login-action@v2.0.0
+        uses: docker/login-action@v3.0.0
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}
           password: ${{ secrets.DOCKERHUB_PASSWORD }}
       - name: Build and push
         id: docker_build
-        uses: docker/build-push-action@v3.0.0
+        uses: docker/build-push-action@v5.2.0
         with:
           push: true
           tags: ${{ secrets.DOCKERHUB_REPOSITORY }}

.github/workflows/pre-commit.yml

@@ -11,6 +11,6 @@ jobs:
     name: pre-commit hooks
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
-      - uses: actions/setup-python@v3
-      - uses: pre-commit/[email protected].0
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5.0.0
+      - uses: pre-commit/[email protected].1

.github/workflows/stale.yml

@@ -7,7 +7,7 @@ jobs:
   stale:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/stale@v5
+      - uses: actions/stale@v9
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           stale-issue-message: "This issue is stale because it has been open 5 months with no activity. Remove stale label or comment or this will be closed in 3 months."

.pre-commit-config.yaml

@@ -10,6 +10,7 @@ repos:
       - id: check-yaml
       - id: debug-statements
       - id: end-of-file-fixer
+        exclude: '.*(\.gitmodules|\.datalad/config|\.gitattributes)$'
       - id: check-executables-have-shebangs
   # Disable until a stable version of type annotation is out.
   # - repo: https://github.com/pre-commit/mirrors-mypy
@@ -28,8 +29,8 @@ repos:
       - id: dockerfilelint
         stages: [commit]
   # Style
-  - repo: https://gitlab.com/pycqa/flake8
-    rev: 3.9.0
+  - repo: https://github.com/pycqa/flake8
+    rev: 6.0.0
     hooks:
       - id: flake8
         additional_dependencies: [flake8-bugbear]

README.md

@@ -2,97 +2,120 @@
 
 [![CircleCI](https://circleci.com/gh/CONP-PCNO/conp-dataset.svg?style=shield)](https://circleci.com/gh/CONP-PCNO/conp-dataset)
 
-CONP dataset is a repository containing the datasets available in the
-Canadian Open Neuroscience Platform. It leverages
-[DataLad](http://datalad.org) to store metadata and references to
-data files distributed in various storage spaces and accessible depending on each data owner's
-policy.
+**"CONP dataset"** is a repository containing the datasets available through the [CONP Portal](https://portal.conp.ca/). It uses [DataLad](http://datalad.org) to store metadata and references to data files distributed in various storage spaces and accessible according to each data owner's policy.
 
-The instructions below explain how to find and get data from the dataset.
-You can also add data by following the instructions in our [contribution
-guidelines](https://github.com/CONP-PCNO/conp-dataset/blob/master/.github/CONTRIBUTING.md).
-We welcome your feedback! :smiley:
+The instructions below explain how to install the necessary software and fetch data from a given dataset. You can also add dataset to the CONP Portal by following the instructions in our [contribution guidelines](https://github.com/CONP-PCNO/conp-dataset/blob/master/.github/CONTRIBUTING.md). We welcome your feedback! :smiley:
 
 ## Dataset structure
 
-`projects` contains sub-datasets for projects.
+Datasets are contained in each directory listed under projects `projects`. Each project maintainer is responsible for the management and curation of their own datasets.
 
-Projects are responsible for the management and curation
-of their own sub-datasets.
+## Installing required software - Method 1: NeuroDebian
 
-## Installing required software
+The following instructions assume a relatively recent Ubuntu- or Debian-based Linux system, though equivalent steps can be taken with other operating systems.
+
+One of the most convenient ways of installing a host of neuro-related software is through the [Neurodebian](https://neuro.debian.net/) repository.  For the following installation procedure, we will assume the use of NeuroDebian, which is installed with:
+
+```
+sudo apt-get update
+sudo apt-get install neurodebian
+```
+
+### Python
+
+The CONP recommends you use Python version 3.12 and up. You can check your Python version with:
+
+```python --version```or ```python3 --version```
 
 ### git
 
-`sudo apt-get install git`
+```sudo apt-get install git```
 
-It is useful to configure your `git` credentials to avoid having to enter them repeatedly:
+It is strongly recommended that you configure your `git` credentials to avoid having to enter them repeatedly or other complications when using DataLad:
 
-`git config --global user.name "yourusername"`
-`git config --global user.email "[email protected]"`
+```git config --global user.name "Jane Doe"```
+
+Replace “John Doe” with your name.
+
+```git config --global user.email “[email protected]”```
+
+Replace “[email protected]” with your email address.
 
 ### git-annex
 
-First install the neurodebian package repository:
+```
+sudo apt-get install git-annex-standalone
+```
 
-`sudo apt-get install neurodebian`
+As of late 2024, this installs git-annex version 10.20241202 or above, which works with CONP datasets. Earlier versions are also compatible but the CONP recommends a relatively recent version for maximum ease. The version of git-annex installed can be verified with:
 
-Then install the version of git-annex included in this repository:
+```git-annex version```
 
-`sudo apt-get install git-annex-standalone`
+### DataLad
 
-The version of git-annex installed can be verified with:
+```sudo apt-get install datalad```
 
-`git annex version`
+## Installing required software - Method 2: `pip`
 
-As of May 12 2020, this installs git annex v 8.20200330, which works with CONP datasets. Earlier versions of git-annex may not.
+The same requirements for `Python` and `git` apply from the previous section.
 
-### DataLad:
+Recent versions of Ubuntu-based distributions (e.g., based on 24.04 LTS) require virtual environments for external Python packages.  You can use `venv` or `pipx` to do this. The easiest for most users is probably to use `pipx`, in which case `git-annex` and `datalad` are installed with the following commands:
 
-`sudo apt-get install datalad`
+```
+pipx install datalad-installer
+datalad-installer git-annex -m datalad/git-annex:release
+git config --global filter.annex.process "git-annex filter-process"
+pipx install datalad
+```
 
-## Getting the data
+## Downloading data
 
 Install the main CONP dataset on your computer:
 
-```console
+```
 datalad install -r http://github.com/CONP-PCNO/conp-dataset
 ```
 
+Install the specific dataset you are interested in:
+
+```
+cd conp-dataset/projects
+datalad install <project name>
+```
+
 Get the files you are interested in:
 
-```console
+```
+cd <project name>
 datalad get <file_name>
 ```
 
-This may require authentication depending on the data owner's configuration.
+This may require authentication depending on the data owner's configuration.  The CONP Portal dataset page corresponding to the dataset you are interested in will indicate whether an account and credentials are required to access data.
 
-You can also search for relevant files and sub-datasets:
+You can also search for relevant files and sub-datasets, for example:
 
-```console
+```
 datalad search T1
 ```
 
 ## Tests
 
-1. Execute `python tests/create_tests.py` from the root of conp-dataset repository
-2. Run `pytest tests/` to execute tests for all datasets in projects and investigators
-3. To run specific test on specific datasets, run `pytest tests/test_<name of dataset>` like
-   `pytest tests/test_projects_SIMON-dataset`
-
 For detailed explanations of the tests, please consult the [test suite documentation](https://github.com/CONP-PCNO/conp-dataset/blob/master/tests/README.md).
 
+1. Execute `python tests/create_tests.py` from the root of conp-dataset repository.
+2. Run `pytest tests/` to execute tests for all datasets in projects and investigators.
+3. To run a specific test on a specific dataset, run `pytest tests/test_<name of dataset>`, e.g., `pytest tests/test_projects_SIMON-dataset`
+
+
 ## Coding standards
 
-To keep the Python code maintainable and readable a suite of QA pipelines is testing the code assuring code standards.
-Pull requests will trigger a GitHub workflow executing pre-commit.
+To keep the Python code maintainable and readable, a suite of quality-assurance pipelines tests the code. Pull requests will trigger a GitHub workflow executing pre-commit.
+
+To execute pre-commit locally, you will need to [install pre-commit](https://pre-commit.com/#installation) using your favorite method. Then, run:
 
-To execute pre-commit locally, you will need to [install pre-commit](https://pre-commit.com/#installation) using your favorite method.
-Then, run:
 ```bash
 pre-commit install
-
 pre-commit run --all-files
 ```
-Pre-commit won't let you commit until reported issue are fixed.
-If problematic, you can optionally skip the pre-commit for a local commit using the `--no-verify` flag when commiting, however this will still perform QA test on your PR.
+
+Pre-commit won't let you commit until reported issues are fixed. If problematic, you can optionally skip the pre-commit for a local commit using the `--no-verify` flag when committing, however this will still perform QA test on your PR.