-
Notifications
You must be signed in to change notification settings - Fork 13
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- change .rst files to .md - update cookiecutter variables - app_name changed to private rendered variable - constants like version numbers set to private unrendered variables - change setup.py to pyproject.toml - updated pipeline_description.json to be more human readible
- Loading branch information
Showing
25 changed files
with
202 additions
and
210 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,10 +1,11 @@ | ||
{ | ||
"full_name": "", | ||
"full_name": "FirstName LastName", | ||
"email": "", | ||
"github_username": "", | ||
"app_full_name": "", | ||
"_output_dir": "", | ||
"app_name": "{{ cookiecutter.app_full_name.lower().replace(' ', '_').replace('-', '_') if cookiecutter.app_full_name else cookiecutter._output_dir }}", | ||
"app_description": "", | ||
"snakebids_version": "0.0.0" | ||
"github": "", | ||
"app_full_name": "Snakebids", | ||
"__app_name": "{{ cookiecutter.app_full_name|lower|replace(' ', '_')|replace('-', '_') }}", | ||
"app_description": "Short application description", | ||
"_app_version": "0.1.0", | ||
"_snakebids_version": "0.0.0", | ||
"_bids_version": "1.8.0" | ||
} |
3 changes: 3 additions & 0 deletions
3
snakebids/project_template/{{ cookiecutter.__app_name }}/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
# {{ cookiecutter.__app_name }} | ||
|
||
{{ cookiecutter.app_description }} |
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
91 changes: 91 additions & 0 deletions
91
...ect_template/{{ cookiecutter.__app_name }}/docs/getting_started/installation.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
## Installation | ||
|
||
Install from github with pip: | ||
|
||
```bash | ||
pip install -e git+https://github.com/{{cookiecutter.github}}/{{cookiecutter.__app_name}}#egg={{cookiecutter.__app_name}} | ||
``` | ||
|
||
Note: you can re-run this command to re-install with the latest version | ||
|
||
## Running the app | ||
|
||
Do a dry-run first (`-n`) and simply print (`-p`) what would be run: | ||
|
||
```bash | ||
{{cookiecutter.__app_name}} /path/to/bids/dir /path/to/output/dir participant -np | ||
``` | ||
|
||
Run the app, using all cores:: | ||
|
||
```bash | ||
{{cookiecutter.__app_name}} /path/to/bids/dir /path/to/output/dir participant --cores all | ||
``` | ||
|
||
If any workflow rules require containers, then run with the `--use-singularity` option. | ||
|
||
## Generating a report | ||
|
||
After your processing is complete, you can use snakemake's `--report` feature to generate | ||
an HTML report. This report will include a graph of all the jobs run, with clickable nodes | ||
to inspect the shell command or python code used in each job, along with the config files and | ||
run times for each job. Workflows may also contain append images for quality assurance or to | ||
summarize outputs, by using the `report(...)` function on any snakemake output. | ||
|
||
To generate a report, run: | ||
|
||
```bash | ||
{{cookiecutter.__app_name}} /path/to/bids/dir /path/to/output/dir participant --report | ||
``` | ||
|
||
## Compute Canada Instructions | ||
|
||
### Setting up a dev environment | ||
|
||
Here are some instructions to get your python environment set-up on graham to run {{ cookiecutter.__app_name }}: | ||
|
||
# Create a virtualenv and activate it: | ||
|
||
```bash | ||
cd $SCRATCH | ||
module load python/3 | ||
virtualenv venv_{{ cookiecutter.__app_name }} | ||
source venv_{{ cookiecutter.__app_name }}/bin/activate | ||
``` | ||
|
||
# Follow the steps above to install from github repository | ||
|
||
### Install job submission helpers | ||
|
||
Snakemake can submit jobs with SLURM, but you need to set-up a Snakemake profile to enable this. The Khan lab has a | ||
snakemake profile that is configured for graham but is customizable upon install, please see `cc-slurm <https://github.com/khanlab/cc-slurm>` for more info. | ||
|
||
If you don't need Snakemake to parallelize jobs across different nodes, you can make use of the simple job submission wrappers in `neuroglia-helpers <https://github.com/khanlab/neuroglia-helpers>`, e.g. `regularSubmit` or `regularInteractive` wrappers. | ||
|
||
These are used in the instructions below. | ||
|
||
### Running jobs on Compute Canada | ||
|
||
In an interactive job (for testing): | ||
|
||
```bash | ||
regularInteractive -n 8 {{ cookiecutter.__app_name }} bids_dir out_dir participant --participant_label 001 -j 8 | ||
``` | ||
|
||
Submitting a job (for larger cores, more subjects), still single job, but snakemake will parallelize over the 32 cores: | ||
|
||
```bash | ||
regularSubmit -j Fat {{ cookiecutter.__app_name }} bids_dir out_dir participant -j 32 | ||
``` | ||
|
||
Scaling up to ~hundred subjects (needs cc-slurm snakemake profile installed), submits 1 16core job per subject: | ||
|
||
```bash | ||
{{ cookiecutter.__app_name }} bids_dir out_dir participant --profile cc-slurm | ||
``` | ||
|
||
Scaling up to even more subjects (uses group-components to bundle multiple subjects in each job), 1 32core job for N subjects (e.g. 10): | ||
|
||
```bash | ||
{{ cookiecutter.__app_name }} bids_dir out_dir participant --profile cc-slurm --group-components subj=10 | ||
``` |
27 changes: 27 additions & 0 deletions
27
snakebids/project_template/{{ cookiecutter.__app_name }}/docs/index.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
```{include} ../README.md | ||
``` | ||
|
||
```{toctree} | ||
:maxdepth: 1 | ||
:caption: Contents: | ||
``` | ||
|
||
```{toctree} | ||
:caption: Getting started | ||
:name: getting_started | ||
:hidden: | ||
:maxdepth: 1 | ||
getting_started/installation | ||
``` | ||
|
||
```{toctree} | ||
:caption: Usage | ||
:name: usage | ||
:hidden: | ||
:maxdepth: 1 | ||
usage/app_cli | ||
usage/snakemake_cli | ||
``` |
4 changes: 4 additions & 0 deletions
4
snakebids/project_template/{{ cookiecutter.__app_name }}/docs/requirements.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
docutils<0.18 | ||
sphinx-argparse | ||
sphinx_rtd_theme | ||
snakebids=={{ cookiecutter._snakebids_version }} |
7 changes: 7 additions & 0 deletions
7
snakebids/project_template/{{ cookiecutter.__app_name }}/docs/usage/app_cli.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
## Command line interface | ||
|
||
```{argparse} | ||
:filename: ../{{ cookiecutter.__app_name }}/run.py | ||
:func: get_parser | ||
:prog: {{ cookiecutter.__app_name }} | ||
``` |
7 changes: 7 additions & 0 deletions
7
...bids/project_template/{{ cookiecutter.__app_name }}/docs/usage/snakemake_cli.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
## Snakemake Command line interface | ||
|
||
```{argparse} | ||
:module: snakemake | ||
:func: get_argument_parser | ||
:prog: snakemake | ||
``` |
19 changes: 19 additions & 0 deletions
19
snakebids/project_template/{{ cookiecutter.__app_name }}/pipeline_description.json
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
{ | ||
"Name": "Dataset generated by {{ cookiecutter.__app_name }}", | ||
"BIDSVersion": "{{ cookiecutter._bids_version }}", | ||
"DatasetType": "derivative", | ||
"GeneratedBy": [ | ||
{ | ||
"Name": "{{ cookiecutter.__app_name }}", | ||
"Version": "{{ cookiecutter._app_version }}"{%- if cookiecutter.github or cookiecutter.full_name or cookiecutter.email -%},{%- endif -%} | ||
{% if cookiecutter.github %} | ||
"CodeURL": "http://github.com/{{ cookiecutter.github }}/{{ cookiecutter.__app_name }}"{%- if cookiecutter.full_name or cookiecutter.email -%},{%- endif -%} | ||
{%- endif -%} | ||
{% if cookiecutter.full_name %} | ||
"Author": "{{ cookiecutter.full_name }}"{%- if cookiecutter.email -%},{%- endif -%} | ||
{%- endif -%} | ||
{% if cookiecutter.email %} | ||
"AuthorEmail": "{{ cookiecutter.email }}"{% endif %} | ||
} | ||
] | ||
} |
29 changes: 29 additions & 0 deletions
29
snakebids/project_template/{{ cookiecutter.__app_name }}/pyproject.toml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
[tool.poetry] | ||
name = "{{ cookiecutter.app_full_name }}" | ||
version = "{{ cookiecutter._app_version }}" | ||
description = "{{ cookiecutter.app_description }}" | ||
authors = [ | ||
"{{ cookiecutter.full_name }}{% if cookiecutter.email %} <{{ cookiecutter.email }}> {% endif %}", | ||
] | ||
readme = "README.md"{% if cookiecutter.github %} | ||
repository = "https://github.com/{{ cookiecutter.github }}/{{ cookiecutter.__app_name }}"{% endif %} | ||
packages = [ | ||
{ include = "{{ cookiecutter.__app_name }}" }, | ||
] | ||
|
||
[tool.poetry.metadata] | ||
classifiers = [ | ||
"Programming Language :: Python :: 3", | ||
"Operating System :: OS Independent" | ||
] | ||
|
||
[tool.poetry.dependencies] | ||
python = ">=3.7, <4.0", | ||
snakebids = ">={{ cookiecutter._snakebids_version }}", | ||
|
||
[tool.poetry.scripts] | ||
{{ cookiecutter.__app_name }} = '{{ cookiecutter.__app_name }}.run:main' | ||
|
||
[build-system] | ||
requires = ["poetry-core>=1.0.0"] | ||
build-backend = "poetry.core.masonry.api"" |
File renamed without changes.
File renamed without changes.
4 changes: 0 additions & 4 deletions
4
snakebids/project_template/{{cookiecutter.app_name}}/README.rst
This file was deleted.
Oops, something went wrong.
86 changes: 0 additions & 86 deletions
86
...roject_template/{{cookiecutter.app_name}}/docs/getting_started/installation.rst
This file was deleted.
Oops, something went wrong.
32 changes: 0 additions & 32 deletions
32
snakebids/project_template/{{cookiecutter.app_name}}/docs/index.rst
This file was deleted.
Oops, something went wrong.
4 changes: 0 additions & 4 deletions
4
snakebids/project_template/{{cookiecutter.app_name}}/docs/requirements.txt
This file was deleted.
Oops, something went wrong.
9 changes: 0 additions & 9 deletions
9
snakebids/project_template/{{cookiecutter.app_name}}/docs/usage/app_cli.rst
This file was deleted.
Oops, something went wrong.
9 changes: 0 additions & 9 deletions
9
snakebids/project_template/{{cookiecutter.app_name}}/docs/usage/snakemake_cli.rst
This file was deleted.
Oops, something went wrong.
11 changes: 0 additions & 11 deletions
11
snakebids/project_template/{{cookiecutter.app_name}}/pipeline_description.json
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.