From 9402f8e3c13f969915778afee4077476179c7fdc Mon Sep 17 00:00:00 2001 From: bendichter Date: Thu, 7 Sep 2023 11:25:30 +0200 Subject: [PATCH 01/53] add docs on the hub --- docs/50_hub.md | 32 ++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 33 insertions(+) create mode 100644 docs/50_hub.md diff --git a/docs/50_hub.md b/docs/50_hub.md new file mode 100644 index 00000000..6ec699fb --- /dev/null +++ b/docs/50_hub.md @@ -0,0 +1,32 @@ +# Using the DANDI Hub + +The DANDI Hub is a JupyterHub instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. + +## Registration + +To use the [DANDI Hub](http://hub.dandiarchive.org), you must first register for an account using the [DANDI website](http://dandiarchive.org). + +## Choosing a server option + +When you start up the DANDI Hub, you will be asked to select across a number of server options. For basic exploration, Tiny or Base would most likely be appropriate. The DANDI Hub also currently offers Medium and Large options, which have more available memory and compute power. The "T4 GPU inference" server comes with an associated T4 GPU, and is intended to be used for applications that require GPU for inference. We request that users of this server be considerate of their usage of the DANDI Hub as a free community resource. Training large deep neural networks is not appropriate. A "Base (MATLAB)" server is also available, which provides a MATLAB cloud installation. + +## Example notebooks + +The best way to share analyses on DANDI data is through the DANDI example notebooks. These notebooks are organized by `///`. Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. + +### Contributing an example notebook + +Notebooks can be submitted as a Pull Request to the [DANDI example-notebooks repository](https://github.com/catalystneuro/example-notebooks). + +#### Environment specification +Best practice is to include one or more notebooks alongside an environment.yml file, which provides a conda-style specification of the environment required to run the notebooks. +1. Create a new environment: `conda create -n -python ` +2. Use `conda install ` and `pip install ` to install the necessary dependencies until the notebook runs through successfully. +3. Export the environment: `conda env export > environment.yml`. + +See detailed instructions for creating a `environment.yml` file [here](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#sharing-an-environment). + +#### File organization +When constructing the Pull Request, ensure that you have the proper directory structure: `///`. If you share more than one notebook, also include a `README.md` file at `///README.md` explaining the purpose of each notebook and providing context and links to relevant publications. + +Once this Pull Requests is accepted, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index f4c50e9f..4f1d5f3a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -34,6 +34,7 @@ nav: - Developer Guide: - Project Structure: "20_project_structure.md" - Notes: "40_development.md" + - DANDI Hub: "50_hub.md" - Terms and Policies: - Terms: "about/terms.md" - Policies: "about/policies.md" From e67b2ccaabb8760cf0c4c5d03e57a5f7801f1672 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Thu, 7 Sep 2023 06:25:35 -0400 Subject: [PATCH 02/53] Update docs/50_hub.md --- docs/50_hub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index 6ec699fb..d12a782a 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -1,6 +1,6 @@ # Using the DANDI Hub -The DANDI Hub is a JupyterHub instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. +[DANDI Hub](http://hub.dandiarchive.org) is a JupyterHub instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. ## Registration From f03a22f0cb6c33ccdfc2eb88a995e796bfa3b618 Mon Sep 17 00:00:00 2001 From: bendichter Date: Thu, 7 Sep 2023 16:52:57 +0200 Subject: [PATCH 03/53] remove specific submission instructions, since they are now a PR on the example-notebooks repo --- docs/50_hub.md | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index 6ec699fb..df3ea70d 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -14,19 +14,4 @@ When you start up the DANDI Hub, you will be asked to select across a number of The best way to share analyses on DANDI data is through the DANDI example notebooks. These notebooks are organized by `///`. Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. -### Contributing an example notebook - -Notebooks can be submitted as a Pull Request to the [DANDI example-notebooks repository](https://github.com/catalystneuro/example-notebooks). - -#### Environment specification -Best practice is to include one or more notebooks alongside an environment.yml file, which provides a conda-style specification of the environment required to run the notebooks. -1. Create a new environment: `conda create -n -python ` -2. Use `conda install ` and `pip install ` to install the necessary dependencies until the notebook runs through successfully. -3. Export the environment: `conda env export > environment.yml`. - -See detailed instructions for creating a `environment.yml` file [here](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#sharing-an-environment). - -#### File organization -When constructing the Pull Request, ensure that you have the proper directory structure: `///`. If you share more than one notebook, also include a `README.md` file at `///README.md` explaining the purpose of each notebook and providing context and links to relevant publications. - Once this Pull Requests is accepted, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file From c4a3d8ce218a519f665cb77645dcc8164037b7f7 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 6 Nov 2023 18:41:46 -0600 Subject: [PATCH 04/53] Added dandi-architecture.drawio --- docs/img/dandi-architecture.drawio | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 docs/img/dandi-architecture.drawio diff --git a/docs/img/dandi-architecture.drawio b/docs/img/dandi-architecture.drawio new file mode 100644 index 00000000..8c8ee028 --- /dev/null +++ b/docs/img/dandi-architecture.drawio @@ -0,0 +1,10 @@ + + + + + + + + + + From 0f359777333ca96bbe4a99a67ff910a80053b64f Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 29 Nov 2023 15:18:07 -0600 Subject: [PATCH 05/53] Add API docs --- mkdocs.yml | 11 +++++++++-- requirements.txt | 1 + 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index f4c50e9f..24b91f93 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -5,7 +5,7 @@ copyright: "CC-BY 4.0" use_directory_urls: true site_url: https://www.dandiarchive.org/handbook/ -# This will use Material them +# Material theme theme: name: "material" language: "en" @@ -31,18 +31,25 @@ nav: - Validation Levels for NWB Files: "135_validation.md" - Publishing Dandisets: "14_publish.md" - Debugging: "15_debugging.md" + - DANDI CLI and Python API Docs: https://dandi.readthedocs.io - Developer Guide: - Project Structure: "20_project_structure.md" - Notes: "40_development.md" + - REST API Swagger Docs: https://api.dandiarchive.org/swagger + - REST API Redoc Docs: https://api.dandiarchive.org/redoc - Terms and Policies: - Terms: "about/terms.md" - Policies: "about/policies.md" - About This Doc: "100_about_this_doc.md" -# list of extension +# List of extensions markdown_extensions: - admonition - pymdownx.details - pymdownx.critic - toc: permalink: True + +# List of plugins +plugins: + - open-in-new-tab diff --git a/requirements.txt b/requirements.txt index 61e03c76..49a1908c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,3 @@ mkdocs-material pymdown-extensions +mkdocs-open-in-new-tab From 273803653c0a9b75a322fc980c29fa5fda91eb91 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 29 Nov 2023 15:20:26 -0600 Subject: [PATCH 06/53] Update text --- mkdocs.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mkdocs.yml b/mkdocs.yml index 24b91f93..820d4315 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -31,12 +31,12 @@ nav: - Validation Levels for NWB Files: "135_validation.md" - Publishing Dandisets: "14_publish.md" - Debugging: "15_debugging.md" - - DANDI CLI and Python API Docs: https://dandi.readthedocs.io + - DANDI CLI and Python API: https://dandi.readthedocs.io - Developer Guide: - Project Structure: "20_project_structure.md" - Notes: "40_development.md" - - REST API Swagger Docs: https://api.dandiarchive.org/swagger - - REST API Redoc Docs: https://api.dandiarchive.org/redoc + - REST API Swagger: https://api.dandiarchive.org/swagger + - REST API Redoc: https://api.dandiarchive.org/redoc - Terms and Policies: - Terms: "about/terms.md" - Policies: "about/policies.md" From 568e2502ade02f61786cced99825019a2a87a835 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 29 Nov 2023 15:23:11 -0600 Subject: [PATCH 07/53] Remove empty test file --- docs/img/dandi-architecture.drawio | 10 ---------- 1 file changed, 10 deletions(-) delete mode 100644 docs/img/dandi-architecture.drawio diff --git a/docs/img/dandi-architecture.drawio b/docs/img/dandi-architecture.drawio deleted file mode 100644 index 8c8ee028..00000000 --- a/docs/img/dandi-architecture.drawio +++ /dev/null @@ -1,10 +0,0 @@ - - - - - - - - - - From 09f9b1befa3a8c951dde7971c572a3e28a096bcc Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 30 Nov 2023 20:40:08 -0600 Subject: [PATCH 08/53] Add search plugin --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index 820d4315..eca2fe2a 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -52,4 +52,5 @@ markdown_extensions: # List of plugins plugins: + - search - open-in-new-tab From d1db03dbb73df8b0ee9a1d38ce3161c126d22f81 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 18 Dec 2023 11:04:39 -0500 Subject: [PATCH 09/53] Update docs/50_hub.md Co-authored-by: Yaroslav Halchenko --- docs/50_hub.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index e7cd7751..72d12aae 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -1,6 +1,7 @@ # Using the DANDI Hub -[DANDI Hub](http://hub.dandiarchive.org) is a JupyterHub instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. +[DANDI Hub](http://hub.dandiarchive.org) is a JupyterHub instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. +Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. ## Registration From 7daa32e53c4caa465d3dcca6b170ed74ce23ddaa Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 18 Dec 2023 11:05:34 -0500 Subject: [PATCH 10/53] Update docs/50_hub.md Co-authored-by: Yaroslav Halchenko --- docs/50_hub.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index 72d12aae..5b72f3e8 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -9,7 +9,13 @@ To use the [DANDI Hub](http://hub.dandiarchive.org), you must first register for ## Choosing a server option -When you start up the DANDI Hub, you will be asked to select across a number of server options. For basic exploration, Tiny or Base would most likely be appropriate. The DANDI Hub also currently offers Medium and Large options, which have more available memory and compute power. The "T4 GPU inference" server comes with an associated T4 GPU, and is intended to be used for applications that require GPU for inference. We request that users of this server be considerate of their usage of the DANDI Hub as a free community resource. Training large deep neural networks is not appropriate. A "Base (MATLAB)" server is also available, which provides a MATLAB cloud installation. +When you start up the DANDI Hub, you will be asked to select across a number of server options. +For basic exploration, Tiny or Base would most likely be appropriate. +The DANDI Hub also currently offers Medium and Large options, which have more available memory and compute power. +The "T4 GPU inference" server comes with an associated T4 GPU, and is intended to be used for applications that require GPU for inference. +We request that users of this server be considerate of their usage of the DANDI Hub as a free community resource. +Training large deep neural networks is not appropriate. +A "Base (MATLAB)" server is also available, which provides a MATLAB cloud installation but you would be required to provide your own license. ## Example notebooks From 9756ff3e2ba46c1cab8ab9429b894df678d14da1 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 18 Dec 2023 11:09:36 -0500 Subject: [PATCH 11/53] Update docs/50_hub.md Co-authored-by: Yaroslav Halchenko --- docs/50_hub.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index 5b72f3e8..a19aefed 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -19,6 +19,8 @@ A "Base (MATLAB)" server is also available, which provides a MATLAB cloud instal ## Example notebooks -The best way to share analyses on DANDI data is through the DANDI example notebooks. These notebooks are organized by `///`. Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. +The best way to share analyses on DANDI data is through the DANDI example notebooks. +These notebooks are maintained in https://github.com/dandi/example-notebooks repository which provides more information about their organization. +Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. Once this Pull Requests is accepted, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file From 933320703c0825b3ddc576417003f6a0b5b851db Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 18 Dec 2023 12:33:29 -0500 Subject: [PATCH 12/53] style - DANDI Archive --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2c02439d..6b0211b5 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ # DANDI Handbook -Handbook for interacting with the DANDI archive. +Handbook for interacting with the DANDI Archive. ## DANDI Style Guidelines @@ -16,6 +16,7 @@ complete word * **license** (**not** licence); in general, prefer American spelling * limited use of "please" * steps should start with 1, not 0 +* **DANDI Archive** - capitalize "archive" if it follows DANDI (**not** DANDI archive) ## HOWTO From 551cbac810c1c4f2036aa3432f791c9aba5afa00 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 23 Dec 2023 15:23:01 -0500 Subject: [PATCH 13/53] Update docs/50_hub.md Co-authored-by: Yaroslav Halchenko --- docs/50_hub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index a19aefed..5e584a9d 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -1,6 +1,6 @@ # Using the DANDI Hub -[DANDI Hub](http://hub.dandiarchive.org) is a JupyterHub instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. +[DANDI Hub](http://hub.dandiarchive.org) is a [JupyterHub](https://jupyterhub.readthedocs.io) instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. ## Registration From e4ea76cdb290fa3e3a8c1bdce304bfb02b8f7b04 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 23 Dec 2023 15:23:35 -0500 Subject: [PATCH 14/53] Update docs/50_hub.md Co-authored-by: Kabilar Gunalan --- docs/50_hub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index 5e584a9d..214cb3ba 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -20,7 +20,7 @@ A "Base (MATLAB)" server is also available, which provides a MATLAB cloud instal ## Example notebooks The best way to share analyses on DANDI data is through the DANDI example notebooks. -These notebooks are maintained in https://github.com/dandi/example-notebooks repository which provides more information about their organization. +These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository which provides more information about their organization. Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. Once this Pull Requests is accepted, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file From d50b4800406483ab983ef81933f4b7074aa3f7a3 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 23 Dec 2023 15:24:03 -0500 Subject: [PATCH 15/53] Update docs/50_hub.md Co-authored-by: Kabilar Gunalan --- docs/50_hub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index 214cb3ba..4530abd0 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -23,4 +23,4 @@ The best way to share analyses on DANDI data is through the DANDI example notebo These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository which provides more information about their organization. Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. -Once this Pull Requests is accepted, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file +Notebooks can be added and updated through a pull request to the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository. Once the pull request is merged, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file From ca4f0640495326e71b3549fb4d1f1a6b15ee7a79 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 9 Jan 2024 09:00:27 -0500 Subject: [PATCH 16/53] Update docs/50_hub.md Co-authored-by: Yaroslav Halchenko --- docs/50_hub.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index 4530abd0..304fd87e 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -23,4 +23,5 @@ The best way to share analyses on DANDI data is through the DANDI example notebo These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository which provides more information about their organization. Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. -Notebooks can be added and updated through a pull request to the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository. Once the pull request is merged, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file +Notebooks can be added and updated through a pull request to the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository. +Once the pull request is merged, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file From 35917e1111d7256a4969f29e1b8a49479034656b Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 9 Jan 2024 09:14:53 -0500 Subject: [PATCH 17/53] Update docs/50_hub.md Co-authored-by: Kabilar Gunalan --- docs/50_hub.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/50_hub.md b/docs/50_hub.md index 304fd87e..4a523f5f 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -1,6 +1,7 @@ # Using the DANDI Hub [DANDI Hub](http://hub.dandiarchive.org) is a [JupyterHub](https://jupyterhub.readthedocs.io) instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI. +For instructions on how to navigate JupyterHub see this [YouTube tutorial](https://www.youtube.com/watch?v=5pf0_bpNbkw&t=09m20s). Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. ## Registration From f4a03f4ce3c66d1ec4560e0cde38e7c01a2ec4c5 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Wed, 17 Jan 2024 22:42:14 -0600 Subject: [PATCH 18/53] Update publish page --- docs/14_publish.md | 36 +++++++++++++++++++++++------------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/14_publish.md b/docs/14_publish.md index 3051d175..aa131482 100644 --- a/docs/14_publish.md +++ b/docs/14_publish.md @@ -1,15 +1,25 @@ # Publishing Dandisets -Once you create a Dandiset, DANDI will automatically create a `draft` version of the Dandiset that -can be -changed as many times as needed. - -Prior to publishing, you should edit your metadata appropriately -(e.g. people and funding contributors, protocol information, keywords, related resources in other -sources such as publications, code repositories, and other data). -If you need to change the Dandiset or its metadata in the future, you will need to create another version of your -Dandiset. The content of the specific published version of the Dandiset cannot be changed. - -When you are ready to publish your data and to create a unique version of your -Dandiset, click the `Publish` button (on the right panel of the landing page). Note that the `Publish` -button is only active if all the metadata and asset errors are fixed. +1. Once you create a Dandiset, DANDI will automatically create a `draft` version + of the Dandiset that can be changed as many times as needed by editing the + metadata or uploading new files. + +1. Prior to publishing, edit the Dandiset metadata. For example, update the +following fields: + 1. People and funding contributors + 1. Protocol information + 1. Keywords + 1. Related resources such as publications and code repositories + +1. When all the metadata and asset errors are fixed, the `Publish` button (on +the right panel of the Dandiset landing page) will be enabled and turn green. + +1. Note, Dandisets with Zarr assets currently cannot be published. We are +actively working on enabling this feature. + +1. Publishing your Dandiset will create a unique version and the content of the +specific published version of the Dandiset cannot be changed. If you need to +change the data or metadata in the future, you will need to create +another version of your Dandiset. + +1. When you are ready to publish your Dandiset, click the `Publish` button. From fc480dcd0f80e2b42f6bb389992b4a433f29cb4f Mon Sep 17 00:00:00 2001 From: Roni Choudhury <2903332+waxlamp@users.noreply.github.com> Date: Wed, 21 Feb 2024 20:29:11 -0500 Subject: [PATCH 19/53] Remix the content a bit This factors some of the general information out of the numbered steps, and includes a bit more detail about what is going on. --- docs/14_publish.md | 46 ++++++++++++++++++++++++++++------------------ 1 file changed, 28 insertions(+), 18 deletions(-) diff --git a/docs/14_publish.md b/docs/14_publish.md index aa131482..cc85c6bd 100644 --- a/docs/14_publish.md +++ b/docs/14_publish.md @@ -1,25 +1,35 @@ # Publishing Dandisets -1. Once you create a Dandiset, DANDI will automatically create a `draft` version - of the Dandiset that can be changed as many times as needed by editing the - metadata or uploading new files. +Once you create a Dandiset, DANDI will automatically create a `draft` version +of the Dandiset that can be changed as many times as needed by editing the +metadata or uploading new files. -1. Prior to publishing, edit the Dandiset metadata. For example, update the -following fields: - 1. People and funding contributors - 1. Protocol information - 1. Keywords - 1. Related resources such as publications and code repositories +When the draft version is ready, you can *publish* your Dandiset. This results +in an immutable snapshot of your Dandiset with its own unique version number +that others can cite. If you need to change the data or metadata, you can do +so by continuing to modify the draft version and publishing a new version +when you are ready. -1. When all the metadata and asset errors are fixed, the `Publish` button (on -the right panel of the Dandiset landing page) will be enabled and turn green. +Follow these steps to publish your Dandiset: -1. Note, Dandisets with Zarr assets currently cannot be published. We are -actively working on enabling this feature. +1. Edit the Dandiset metadata, aiming to fix all Dandiset metadata validation + errors, and include any other useful information. For example, you may want + to edit the following fields: + - People and funding contributors + - Protocol information + - Keywords + - Related resources such as publications and code repositories + +1. Fix all asset metadata errors by modifying the asset files to eliminate + the errors and re-uploading them. -1. Publishing your Dandiset will create a unique version and the content of the -specific published version of the Dandiset cannot be changed. If you need to -change the data or metadata in the future, you will need to create -another version of your Dandiset. +1. When all the Dandiset metadata and asset metadata errors are fixed, the + `Publish` button (on the right panel of the Dandiset landing page) will + be enabled and turn green. Click the button to publish your Dandiset. -1. When you are ready to publish your Dandiset, click the `Publish` button. +1. In the lower right section of the Dandiset landing page, you should see + the new, published version of your Dandiset listed. Click on that link + to view this version. + +**NOTE:** Dandisets with Zarr assets currently cannot be published. We are +actively working on enabling this feature. From 5a88a812e84e51a1fcf8034c2cadc519845859e8 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 12:03:56 -0600 Subject: [PATCH 20/53] Update homepage link for the logo in the header --- mkdocs.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 10211565..526906ac 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -55,3 +55,7 @@ markdown_extensions: plugins: - search - open-in-new-tab + +# Customize theme +extra: + homepage: https://dandiarchive.org From ed0c3524d87be2aea210fd99184f482442845948 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 12:17:12 -0600 Subject: [PATCH 21/53] Update DOI for DANDI CLI to cite all versions --- docs/10_using_dandi.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/10_using_dandi.md b/docs/10_using_dandi.md index 864d9e14..7e259a46 100644 --- a/docs/10_using_dandi.md +++ b/docs/10_using_dandi.md @@ -108,7 +108,7 @@ Note that `Dandihub` is not intended for significant computation, but provides a You can add the following statement to the methods section of your manuscript. > Data and associated metadata were uploaded to the DANDI archive [RRID:SCR_017571] using - the Python command line tool (https://doi.org/10.5281/zenodo.7041535). The data were first + the Python command line tool (https://doi.org/10.5281/zenodo.3692138). The data were first converted into the NWB format (https://doi.org/10.1101/2021.03.13.435173) and organized into a BIDS-like (https://doi.org/10.1038/sdata.2016.44) structure. @@ -116,4 +116,4 @@ You can refer to DANDI using any of the following options: * Using an RRID [RRID:SCR_017571](https://scicrunch.org/scicrunch/Resources/record/nlx_144509-1/SCR_017571/resolver). -* Using the DANDI CLI reference: https://doi.org/10.5281/zenodo.7041535 +* Using the DANDI CLI reference: https://doi.org/10.5281/zenodo.3692138 From d9f677305d00c9be5f6d073da09453e31286b4cc Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 14:02:03 -0600 Subject: [PATCH 22/53] Add markdown extension for auto-linking http links --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index 10211565..0bc65e7f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -48,6 +48,7 @@ markdown_extensions: - admonition - pymdownx.details - pymdownx.critic + - pymdownx.magiclink - toc: permalink: True From 87c8490056304e6e9b8eaab48cf919ec2228dba5 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 14:05:37 -0600 Subject: [PATCH 23/53] Remove markdown formatted links --- docs/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/index.md b/docs/index.md index ac520be6..10966d24 100644 --- a/docs/index.md +++ b/docs/index.md @@ -5,7 +5,7 @@ src="./img/dandi-banner.png" alt="dandi_banner" style="width: 75%; height: auto; display: block; margin-left: auto; margin-right: auto;"/> -The Web interface to the DANDI archive is located at [https://dandiarchive.org](https://dandiarchive.org). +The Web interface to the DANDI archive is located at https://dandiarchive.org. This documentation explains how to interact with the archive. ## How to Use This Documentation @@ -26,7 +26,7 @@ You can communicate with the DANDI team in a variety of ways, depending on your - You can ask questions, report bugs, or request features [at our helpdesk](https://github.com/dandi/helpdesk/issues/new/choose). -- For interacting with the global neuroscience community, post on [https://neurostars.org](https://neurostars.org) +- For interacting with the global neuroscience community, post on https://neurostars.org and use the tag [dandi](https://neurostars.org/tag/dandi). - You can use the DANDI Slack workspace, which we will invite you to after approving your [registration on DANDI using GitHub](https://dandiarchive.org/) (this registration is required to upload data or to use the DANDI From a71bc1d24ebe289f6fbbd8f0171da6ecede7b450 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 14:39:14 -0600 Subject: [PATCH 24/53] Add `Create a DANDI Account` page --- docs/13_upload.md | 12 +----------- docs/16_account.md | 13 +++++++++++++ docs/50_hub.md | 1 + mkdocs.yml | 1 + 4 files changed, 16 insertions(+), 11 deletions(-) create mode 100644 docs/16_account.md diff --git a/docs/13_upload.md b/docs/13_upload.md index cdafa31e..2d48d174 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -2,17 +2,6 @@ To create a new Dandiset and upload your data, you need to have a DANDI account. -## Create an Account on DANDI - -To create a DANDI account: - -1. [Create a GitHub account](https://github.com/) if you don't have one. -2. Using your GitHub account, [register a DANDI account](https://gui.dandiarchive.org/#/user/register). - -You will receive an email acknowledging activation of your account within 24 -hours, after which you can log in to DANDI using GitHub by clicking the login -button. - ## Create a Dandiset and Add Data You can create a new Dandiset at [http://dandiarchive.org](http://dandiarchive.org). This Dandiset can be fully @@ -30,6 +19,7 @@ two different servers differ slightly. ### **Setup** +1. To create a new Dandiset and upload your data, you need to have a DANDI account. See [Create a DANDI Account](./16_account.md). 1. Log in to DANDI and copy your API key. Click on your user initials in the top-right corner after logging in. Production (dandiarchive.org) and staging (gui-staging.dandiarchive.org) servers have different API keys and different logins. diff --git a/docs/16_account.md b/docs/16_account.md new file mode 100644 index 00000000..79b751da --- /dev/null +++ b/docs/16_account.md @@ -0,0 +1,13 @@ +# Create a DANDI Account + +A DANDI account is only required for specific features. +Without a DANDI account, you can search, view, and download files. +With a DANDI account, you can additionally create Dandisets and access the DANDI Hub to analyze existing data. + +## Instructions + +1. To create a DANDI account, first [create a GitHub account](https://github.com/) if you don't have one. +1. Using your GitHub account, register for a DANDI account by selecting the `LOG IN WITH GITHUB` button on the [DANDI homepage](https://dandiarchive.org). +1. You will receive an email acknowledging activation of your account within 24 +hours, after which you can log in to DANDI using GitHub by clicking the +`LOG IN WITH GITHUB` button. \ No newline at end of file diff --git a/docs/50_hub.md b/docs/50_hub.md index 4a523f5f..17b9a32f 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -7,6 +7,7 @@ Note that DANDI Hub is not intended for significant computation, but provides a ## Registration To use the [DANDI Hub](http://hub.dandiarchive.org), you must first register for an account using the [DANDI website](http://dandiarchive.org). +See [Create a DANDI Account](./16_account.md). ## Choosing a server option diff --git a/mkdocs.yml b/mkdocs.yml index 10211565..8ba59a91 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -25,6 +25,7 @@ nav: - Data Licenses: "35_data_licenses.md" - User Guide: - Using DANDI: "10_using_dandi.md" + - Create DANDI Account: "16_account.md" - Viewing Dandisets: "11_view.md" - Downloading Data and Dandisets: "12_download.md" - Creating Dandisets and Uploading Data: "13_upload.md" From c4ef796b376dc0403389ceefbaa8d9d89b45909f Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 14:42:08 -0600 Subject: [PATCH 25/53] Update text --- docs/13_upload.md | 2 +- docs/50_hub.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index 2d48d174..62cdafe3 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -19,7 +19,7 @@ two different servers differ slightly. ### **Setup** -1. To create a new Dandiset and upload your data, you need to have a DANDI account. See [Create a DANDI Account](./16_account.md). +1. To create a new Dandiset and upload your data, you need to have a DANDI account. See the [Create a DANDI Account](./16_account.md) page. 1. Log in to DANDI and copy your API key. Click on your user initials in the top-right corner after logging in. Production (dandiarchive.org) and staging (gui-staging.dandiarchive.org) servers have different API keys and different logins. diff --git a/docs/50_hub.md b/docs/50_hub.md index 17b9a32f..b817ec94 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -7,7 +7,7 @@ Note that DANDI Hub is not intended for significant computation, but provides a ## Registration To use the [DANDI Hub](http://hub.dandiarchive.org), you must first register for an account using the [DANDI website](http://dandiarchive.org). -See [Create a DANDI Account](./16_account.md). +See the [Create a DANDI Account](./16_account.md) page. ## Choosing a server option From b197b7e373dfdc5198f1ea7747a979c640624af7 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 14:48:56 -0600 Subject: [PATCH 26/53] Remove markdown formatted links --- docs/13_upload.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index cdafa31e..72062ed0 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -15,12 +15,12 @@ button. ## Create a Dandiset and Add Data -You can create a new Dandiset at [http://dandiarchive.org](http://dandiarchive.org). This Dandiset can be fully +You can create a new Dandiset at https://dandiarchive.org. This Dandiset can be fully public or embargoed according to NIH policy. When you create a Dandiset, a permanent ID is automatically assigned to it. To prevent the production server from being inundated with test Dandisets, we encourage developers to develop -against the development server ([https://gui-staging.dandiarchive.org/](https://gui-staging.dandiarchive.org/)). Note +against the development server (https://gui-staging.dandiarchive.org/). Note that the development server should not be used to stage your data. All data are uploaded as draft and can be adjusted before publishing on the production server. The development server is primarily used by users learning to use DANDI or by developers. From 5237a4a38313155aa3119c469b09de10b094e02a Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 14:53:22 -0600 Subject: [PATCH 27/53] Add social icons to footer --- mkdocs.yml | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 10211565..9bf11456 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -55,3 +55,22 @@ markdown_extensions: plugins: - search - open-in-new-tab + +# Customize theme +extra: + social: + - icon: fontawesome/brands/slack + link: https://dandiarchive.slack.com + name: Slack + - icon: fontawesome/solid/paper-plane + link: mailto:info@dandiarchive.org + name: Email + - icon: fontawesome/brands/x-twitter + link: https://twitter.com/dandiarchive + name: X-Twitter + - icon: fontawesome/brands/github + link: https://github.com/dandi + name: GitHub + - icon: fontawesome/brands/youtube + link: https://www.youtube.com/@dandiarchive + name: YouTube \ No newline at end of file From e0beb1910bbbf870fd97824e24833d0048f3fce2 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 15:10:22 -0600 Subject: [PATCH 28/53] Add headings for each option to store access credentials --- docs/13_upload.md | 87 +++++++++++++++++++++++++---------------------- 1 file changed, 46 insertions(+), 41 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index cdafa31e..f205c5e2 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -99,44 +99,49 @@ two different servers differ slightly. If you have an issue using the Python CLI, see the [Dandi Debugging section](./15_debugging.md). -## Storing Access Credentials - -By default, the DANDI CLI looks for an API key in the `DANDI_API_KEY` -environment variable. To set this on Linux or macOS, run - -```bash -export DANDI_API_KEY=personal-key-value -``` -*Note that there are no spaces around the "=". - -If this is not set, the CLI will look up the API -key using the [keyring](https://github.com/jaraco/keyring) library, which -supports numerous backends, including the system keyring, an encrypted keyfile, -and a plaintext (unencrypted) keyfile. - -- You can store your API key where the `keyring` library can find it by using - the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the - API key when asked for the password for `key` in `dandi-api-dandi`. - -- You can set the backend the `keyring` library uses either by setting the - `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring` - library's configuration file](https://github.com/jaraco/keyring#configuring). - IDs for the available backends can be listed by running `keyring --list`. If - no backend is specified in this way, the library will use the available - backend with the highest priority. - -If the API key isn't stored in either the `DANDI_API_KEY` environment variable -or in the keyring, the CLI will prompt you to enter the API key, and then it -will store it in the keyring. This may cause you to be prompted further; you -may be asked to enter a password to encrypt/decrypt the keyring, or you may be -asked by your OS to confirm whether to give the DANDI CLI access to the -keyring. - -- If the DANDI CLI encounters an error while attempting to fetch the API key - from the default keyring backend, it will fall back to using an encrypted - keyfile (the `keyrings.alt.file.EncryptedKeyring` backend). If the keyfile - does not already exist, the CLI will ask you for confirmation; if you answer - "yes," the `keyring` configuration file (if it does not already exist; see - above) will be configured to use `EncryptedKeyring` as the default backend. - If you answer "no," the CLI will exit with an error, and you must store the - API key somewhere accessible to the CLI on your own. +## Store Access Credentials + +There are three options for storing your DANDI access credentials. + +1. `DANDI_API_KEY` Environment Variable + + - By default, the DANDI CLI looks for an API key in the `DANDI_API_KEY` + environment variable. To set this on Linux or macOS, run: + + export DANDI_API_KEY=personal-key-value + + - Note that there are no spaces around the "=". + +1. `keyring` Library + - If the `DANDI_API_KEY` environment variable is not set, the CLI will look up the API + key using the [keyring](https://github.com/jaraco/keyring) library, which + supports numerous backends, including the system keyring, an encrypted keyfile, + and a plaintext (unencrypted) keyfile. + + - You can store your API key where the `keyring` library can find it by using + the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the + API key when asked for the password for `key` in `dandi-api-dandi`. + + - You can set the backend the `keyring` library uses either by setting the + `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring` + library's configuration file](https://github.com/jaraco/keyring#configuring). + IDs for the available backends can be listed by running `keyring --list`. If + no backend is specified in this way, the library will use the available + backend with the highest priority. + +1. Manual Password Entry + - If the API key isn't stored in either the `DANDI_API_KEY` environment variable + or in the keyring, the CLI will prompt you to enter the API key, and then it + will store it in the keyring. This may cause you to be prompted further; you + may be asked to enter a password to encrypt/decrypt the keyring, or you may be + asked by your OS to confirm whether to give the DANDI CLI access to the + keyring. + + - If the DANDI CLI encounters an error while attempting to fetch the API key + from the default keyring backend, it will fall back to using an encrypted + keyfile (the `keyrings.alt.file.EncryptedKeyring` backend). If the keyfile + does not already exist, the CLI will ask you for confirmation; if you answer + "yes," the `keyring` configuration file (if it does not already exist; see + above) will be configured to use `EncryptedKeyring` as the default backend. + If you answer "no," the CLI will exit with an error, and you must store the + API key somewhere accessible to the CLI on your own. From a66d91b83887f8c0b604ebb646577ba8c0c58876 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Thu, 22 Feb 2024 15:14:26 -0600 Subject: [PATCH 29/53] Revert imperative-style change --- docs/13_upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index f205c5e2..eeeb005f 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -99,7 +99,7 @@ two different servers differ slightly. If you have an issue using the Python CLI, see the [Dandi Debugging section](./15_debugging.md). -## Store Access Credentials +## Storing Access Credentials There are three options for storing your DANDI access credentials. From e955b29e0940e96fff3e0455589f6475a924d5b3 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 11:40:05 -0600 Subject: [PATCH 30/53] Add GitHub Action to test the build of mkdocs site --- .github/workflows/build.yaml | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 .github/workflows/build.yaml diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml new file mode 100644 index 00000000..df5c6f48 --- /dev/null +++ b/.github/workflows/build.yaml @@ -0,0 +1,22 @@ +name: Build MkDocs +on: + push: + pull_request: + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Install version of Python + uses: actions/setup-python@v5 + with: + python-version: '3.10' + + - name: Install requirements + run: pip install -r requirements.txt + + - name: Build docs + run: mkdocs build From ced89c24f76e68c8a2057a8f340fc6df26f9cd32 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 11:55:02 -0600 Subject: [PATCH 31/53] Update Python versoin --- .github/workflows/build.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index df5c6f48..c6fa2f84 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -13,7 +13,7 @@ jobs: - name: Install version of Python uses: actions/setup-python@v5 with: - python-version: '3.10' + python-version: '3.11' - name: Install requirements run: pip install -r requirements.txt From 02974ca982d5cd28f7ebb86c9b7ea9a19e881025 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 12:21:54 -0600 Subject: [PATCH 32/53] Rename file --- .github/workflows/{build.yaml => build-docs.yaml} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename .github/workflows/{build.yaml => build-docs.yaml} (100%) diff --git a/.github/workflows/build.yaml b/.github/workflows/build-docs.yaml similarity index 100% rename from .github/workflows/build.yaml rename to .github/workflows/build-docs.yaml From ed68af8f4298225399bcbd27d798d11df9655300 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 12:22:19 -0600 Subject: [PATCH 33/53] Revert actions/checkout version --- .github/workflows/build-docs.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/build-docs.yaml b/.github/workflows/build-docs.yaml index c6fa2f84..25b43b95 100644 --- a/.github/workflows/build-docs.yaml +++ b/.github/workflows/build-docs.yaml @@ -8,7 +8,7 @@ jobs: runs-on: ubuntu-latest steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@v1 - name: Install version of Python uses: actions/setup-python@v5 From 668cc67d8fdaa46f8dd3b29b518660e5fbb6c2b8 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 12:42:48 -0600 Subject: [PATCH 34/53] Pin `mkdocs-material` version to force install later version --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 49a1908c..1952f709 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,3 @@ -mkdocs-material +mkdocs-material>=9.5.10 pymdown-extensions mkdocs-open-in-new-tab From 9672a4144534e814a0e65d6d1f6b0e1e77313280 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 15:49:11 -0600 Subject: [PATCH 35/53] Update docs/14_publish.md Co-authored-by: Yaroslav Halchenko --- docs/14_publish.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/14_publish.md b/docs/14_publish.md index cc85c6bd..e2baf867 100644 --- a/docs/14_publish.md +++ b/docs/14_publish.md @@ -23,7 +23,7 @@ Follow these steps to publish your Dandiset: 1. Fix all asset metadata errors by modifying the asset files to eliminate the errors and re-uploading them. -1. When all the Dandiset metadata and asset metadata errors are fixed, the +1. When all the Dandiset metadata and asset metadata errors are fixed, and the dandiset made public if it was initially embargoed, the `Publish` button (on the right panel of the Dandiset landing page) will be enabled and turn green. Click the button to publish your Dandiset. From 51114fc2411efbb3a71a3aa76029ee58dd1cbeda Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 15:50:10 -0600 Subject: [PATCH 36/53] Update docs/14_publish.md --- docs/14_publish.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/14_publish.md b/docs/14_publish.md index e2baf867..d5f84f9a 100644 --- a/docs/14_publish.md +++ b/docs/14_publish.md @@ -23,7 +23,7 @@ Follow these steps to publish your Dandiset: 1. Fix all asset metadata errors by modifying the asset files to eliminate the errors and re-uploading them. -1. When all the Dandiset metadata and asset metadata errors are fixed, and the dandiset made public if it was initially embargoed, the +1. When all the Dandiset metadata and asset metadata errors are fixed, and the Dandiset is made public if it was initially embargoed, the `Publish` button (on the right panel of the Dandiset landing page) will be enabled and turn green. Click the button to publish your Dandiset. From 9f0d2ca3a099eff5d2a1223a9ac99b24b08235be Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 16:20:52 -0600 Subject: [PATCH 37/53] Update docs/16_account.md Co-authored-by: Yaroslav Halchenko --- docs/16_account.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/16_account.md b/docs/16_account.md index 79b751da..f5b2bd8c 100644 --- a/docs/16_account.md +++ b/docs/16_account.md @@ -8,6 +8,6 @@ With a DANDI account, you can additionally create Dandisets and access the DANDI 1. To create a DANDI account, first [create a GitHub account](https://github.com/) if you don't have one. 1. Using your GitHub account, register for a DANDI account by selecting the `LOG IN WITH GITHUB` button on the [DANDI homepage](https://dandiarchive.org). -1. You will receive an email acknowledging activation of your account within 24 -hours, after which you can log in to DANDI using GitHub by clicking the -`LOG IN WITH GITHUB` button. \ No newline at end of file +1. You will receive an email acknowledging that your request for an account will be reviewed within 24 hours: + - **note**: requests from a new GitHub accounts and from emails not withing `.edu` domain might take longer to review and more likely to be rejected, especially if there are no plans described to upload data to the archive +1. If your request for an account approved, you would be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button. \ No newline at end of file From d3bb3b6dbcbda2f49f9a6687989883b95f2fb472 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 16:21:16 -0600 Subject: [PATCH 38/53] Update docs/16_account.md --- docs/16_account.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/16_account.md b/docs/16_account.md index f5b2bd8c..73c6a910 100644 --- a/docs/16_account.md +++ b/docs/16_account.md @@ -8,6 +8,6 @@ With a DANDI account, you can additionally create Dandisets and access the DANDI 1. To create a DANDI account, first [create a GitHub account](https://github.com/) if you don't have one. 1. Using your GitHub account, register for a DANDI account by selecting the `LOG IN WITH GITHUB` button on the [DANDI homepage](https://dandiarchive.org). -1. You will receive an email acknowledging that your request for an account will be reviewed within 24 hours: +1. You will receive an email acknowledging that your request for an account will be reviewed within 24 hours. - **note**: requests from a new GitHub accounts and from emails not withing `.edu` domain might take longer to review and more likely to be rejected, especially if there are no plans described to upload data to the archive 1. If your request for an account approved, you would be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button. \ No newline at end of file From 294ee22179aa5b30e96c974edd53faffc1bef463 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 16:22:12 -0600 Subject: [PATCH 39/53] Update docs/16_account.md --- docs/16_account.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/16_account.md b/docs/16_account.md index 73c6a910..13a57d5c 100644 --- a/docs/16_account.md +++ b/docs/16_account.md @@ -10,4 +10,4 @@ With a DANDI account, you can additionally create Dandisets and access the DANDI 1. Using your GitHub account, register for a DANDI account by selecting the `LOG IN WITH GITHUB` button on the [DANDI homepage](https://dandiarchive.org). 1. You will receive an email acknowledging that your request for an account will be reviewed within 24 hours. - **note**: requests from a new GitHub accounts and from emails not withing `.edu` domain might take longer to review and more likely to be rejected, especially if there are no plans described to upload data to the archive -1. If your request for an account approved, you would be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button. \ No newline at end of file +1. If your request for an account is approved, you will be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button. \ No newline at end of file From 4e29b82f0711a2362d68e47ba6eccc9728ba9aa2 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 16:32:50 -0600 Subject: [PATCH 40/53] Update docs/16_account.md --- docs/16_account.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/16_account.md b/docs/16_account.md index 13a57d5c..61aac25d 100644 --- a/docs/16_account.md +++ b/docs/16_account.md @@ -9,5 +9,5 @@ With a DANDI account, you can additionally create Dandisets and access the DANDI 1. To create a DANDI account, first [create a GitHub account](https://github.com/) if you don't have one. 1. Using your GitHub account, register for a DANDI account by selecting the `LOG IN WITH GITHUB` button on the [DANDI homepage](https://dandiarchive.org). 1. You will receive an email acknowledging that your request for an account will be reviewed within 24 hours. - - **note**: requests from a new GitHub accounts and from emails not withing `.edu` domain might take longer to review and more likely to be rejected, especially if there are no plans described to upload data to the archive + - **note**: Requests from new GitHub accounts and from emails that are not an `.edu` domain might take longer to review and are more likely to be rejected, especially if there are no plans described to upload data to the archive. 1. If your request for an account is approved, you will be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button. \ No newline at end of file From f55eec6d0dfb89cc1e8182bdda49c2eb994351c3 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 23:20:38 -0600 Subject: [PATCH 41/53] Change order of instructions --- docs/13_upload.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index eeeb005f..a89101ab 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -101,7 +101,7 @@ If you have an issue using the Python CLI, see the [Dandi Debugging section](./1 ## Storing Access Credentials -There are three options for storing your DANDI access credentials. +There are two options for storing your DANDI access credentials. 1. `DANDI_API_KEY` Environment Variable @@ -113,30 +113,30 @@ There are three options for storing your DANDI access credentials. - Note that there are no spaces around the "=". 1. `keyring` Library - - If the `DANDI_API_KEY` environment variable is not set, the CLI will look up the API + 1. If the `DANDI_API_KEY` environment variable is not set, the CLI will look up the API key using the [keyring](https://github.com/jaraco/keyring) library, which supports numerous backends, including the system keyring, an encrypted keyfile, and a plaintext (unencrypted) keyfile. - - You can store your API key where the `keyring` library can find it by using + 2. Methods to store the API key with `keyring` + 1. You can store your API key where the `keyring` library can find it by using the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the API key when asked for the password for `key` in `dandi-api-dandi`. - - You can set the backend the `keyring` library uses either by setting the - `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring` - library's configuration file](https://github.com/jaraco/keyring#configuring). - IDs for the available backends can be listed by running `keyring --list`. If - no backend is specified in this way, the library will use the available - backend with the highest priority. - -1. Manual Password Entry - - If the API key isn't stored in either the `DANDI_API_KEY` environment variable + 2. If the API key isn't stored in either the `DANDI_API_KEY` environment variable or in the keyring, the CLI will prompt you to enter the API key, and then it will store it in the keyring. This may cause you to be prompted further; you may be asked to enter a password to encrypt/decrypt the keyring, or you may be asked by your OS to confirm whether to give the DANDI CLI access to the keyring. + 3. `keyring` Backend + - You can set the backend the `keyring` library uses either by setting the + `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring` + library's configuration file](https://github.com/jaraco/keyring#configuring). + - IDs for the available backends can be listed by running `keyring --list`. + - If no backend is specified in this way, the library will use the available + backend with the highest priority. - If the DANDI CLI encounters an error while attempting to fetch the API key from the default keyring backend, it will fall back to using an encrypted keyfile (the `keyrings.alt.file.EncryptedKeyring` backend). If the keyfile From f8da6ccdbe1b99bd19543251a9502e04523dec6a Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 23:26:31 -0600 Subject: [PATCH 42/53] Fix indentation --- docs/13_upload.md | 64 +++++++++++++++++++++++------------------------ 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index a89101ab..cf851a6d 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -113,35 +113,35 @@ There are two options for storing your DANDI access credentials. - Note that there are no spaces around the "=". 1. `keyring` Library - 1. If the `DANDI_API_KEY` environment variable is not set, the CLI will look up the API - key using the [keyring](https://github.com/jaraco/keyring) library, which - supports numerous backends, including the system keyring, an encrypted keyfile, - and a plaintext (unencrypted) keyfile. - - 2. Methods to store the API key with `keyring` - 1. You can store your API key where the `keyring` library can find it by using - the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the - API key when asked for the password for `key` in `dandi-api-dandi`. - - 2. If the API key isn't stored in either the `DANDI_API_KEY` environment variable - or in the keyring, the CLI will prompt you to enter the API key, and then it - will store it in the keyring. This may cause you to be prompted further; you - may be asked to enter a password to encrypt/decrypt the keyring, or you may be - asked by your OS to confirm whether to give the DANDI CLI access to the - keyring. - - 3. `keyring` Backend - - You can set the backend the `keyring` library uses either by setting the - `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring` - library's configuration file](https://github.com/jaraco/keyring#configuring). - - IDs for the available backends can be listed by running `keyring --list`. - - If no backend is specified in this way, the library will use the available - backend with the highest priority. - - If the DANDI CLI encounters an error while attempting to fetch the API key - from the default keyring backend, it will fall back to using an encrypted - keyfile (the `keyrings.alt.file.EncryptedKeyring` backend). If the keyfile - does not already exist, the CLI will ask you for confirmation; if you answer - "yes," the `keyring` configuration file (if it does not already exist; see - above) will be configured to use `EncryptedKeyring` as the default backend. - If you answer "no," the CLI will exit with an error, and you must store the - API key somewhere accessible to the CLI on your own. + - If the `DANDI_API_KEY` environment variable is not set, the CLI will look up the API + key using the [keyring](https://github.com/jaraco/keyring) library, which + supports numerous backends, including the system keyring, an encrypted keyfile, + and a plaintext (unencrypted) keyfile. + + - Methods to store the API key with `keyring` + 1. You can store your API key where the `keyring` library can find it by using + the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the + API key when asked for the password for `key` in `dandi-api-dandi`. + + 2. If the API key isn't stored in either the `DANDI_API_KEY` environment variable + or in the keyring, the CLI will prompt you to enter the API key, and then it + will store it in the keyring. This may cause you to be prompted further; you + may be asked to enter a password to encrypt/decrypt the keyring, or you may be + asked by your OS to confirm whether to give the DANDI CLI access to the + keyring. + + - `keyring` Backend + - You can set the backend the `keyring` library uses either by setting the + `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring` + library's configuration file](https://github.com/jaraco/keyring#configuring). + - IDs for the available backends can be listed by running `keyring --list`. + - If no backend is specified in this way, the library will use the available + backend with the highest priority. + - If the DANDI CLI encounters an error while attempting to fetch the API key + from the default keyring backend, it will fall back to using an encrypted + keyfile (the `keyrings.alt.file.EncryptedKeyring` backend). If the keyfile + does not already exist, the CLI will ask you for confirmation; if you answer + "yes," the `keyring` configuration file (if it does not already exist; see + above) will be configured to use `EncryptedKeyring` as the default backend. + If you answer "no," the CLI will exit with an error, and you must store the + API key somewhere accessible to the CLI on your own. From 0397d054eddab1230541ff66deb70d8f219cfdcd Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 23:27:19 -0600 Subject: [PATCH 43/53] Fix numbering --- docs/13_upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index cf851a6d..fb93f4cc 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -112,7 +112,7 @@ There are two options for storing your DANDI access credentials. - Note that there are no spaces around the "=". -1. `keyring` Library +2. `keyring` Library - If the `DANDI_API_KEY` environment variable is not set, the CLI will look up the API key using the [keyring](https://github.com/jaraco/keyring) library, which supports numerous backends, including the system keyring, an encrypted keyfile, From 989295cec6737892c947da22dcd02e00870de20b Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 23:34:55 -0600 Subject: [PATCH 44/53] Reorder instructions --- docs/13_upload.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index fb93f4cc..93e5833e 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -118,19 +118,7 @@ There are two options for storing your DANDI access credentials. supports numerous backends, including the system keyring, an encrypted keyfile, and a plaintext (unencrypted) keyfile. - - Methods to store the API key with `keyring` - 1. You can store your API key where the `keyring` library can find it by using - the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the - API key when asked for the password for `key` in `dandi-api-dandi`. - - 2. If the API key isn't stored in either the `DANDI_API_KEY` environment variable - or in the keyring, the CLI will prompt you to enter the API key, and then it - will store it in the keyring. This may cause you to be prompted further; you - may be asked to enter a password to encrypt/decrypt the keyring, or you may be - asked by your OS to confirm whether to give the DANDI CLI access to the - keyring. - - - `keyring` Backend + - Specifying the `keyring` backend - You can set the backend the `keyring` library uses either by setting the `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring` library's configuration file](https://github.com/jaraco/keyring#configuring). @@ -145,3 +133,15 @@ There are two options for storing your DANDI access credentials. above) will be configured to use `EncryptedKeyring` as the default backend. If you answer "no," the CLI will exit with an error, and you must store the API key somewhere accessible to the CLI on your own. + + - Methods to store the API key with `keyring` + 1. You can store your API key where the `keyring` library can find it by using + the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the + API key when asked for the password for `key` in `dandi-api-dandi`. + + 2. If the API key isn't stored in either the `DANDI_API_KEY` environment variable + or in the keyring, the CLI will prompt you to enter the API key, and then it + will store it in the keyring. This may cause you to be prompted further; you + may be asked to enter a password to encrypt/decrypt the keyring, or you may be + asked by your OS to confirm whether to give the DANDI CLI access to the + keyring. From 13c833f36741289f51e80af0a211c8734692a657 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 23:36:45 -0600 Subject: [PATCH 45/53] Update title --- docs/13_upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index 93e5833e..81d4a3ce 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -134,7 +134,7 @@ There are two options for storing your DANDI access credentials. If you answer "no," the CLI will exit with an error, and you must store the API key somewhere accessible to the CLI on your own. - - Methods to store the API key with `keyring` + - Storing the API key with `keyring` 1. You can store your API key where the `keyring` library can find it by using the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the API key when asked for the password for `key` in `dandi-api-dandi`. From 93935365584e1bc028470d75370142e1dad6febb Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Fri, 23 Feb 2024 23:44:26 -0600 Subject: [PATCH 46/53] Fix formatting and syntax --- docs/13_upload.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/13_upload.md b/docs/13_upload.md index 81d4a3ce..47f922a3 100644 --- a/docs/13_upload.md +++ b/docs/13_upload.md @@ -120,8 +120,8 @@ There are two options for storing your DANDI access credentials. - Specifying the `keyring` backend - You can set the backend the `keyring` library uses either by setting the - `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring` - library's configuration file](https://github.com/jaraco/keyring#configuring). + `PYTHON_KEYRING_BACKEND` environment variable or by filling in the `keyring` + library's [configuration file](https://github.com/jaraco/keyring#configuring). - IDs for the available backends can be listed by running `keyring --list`. - If no backend is specified in this way, the library will use the available backend with the highest priority. @@ -143,5 +143,5 @@ There are two options for storing your DANDI access credentials. or in the keyring, the CLI will prompt you to enter the API key, and then it will store it in the keyring. This may cause you to be prompted further; you may be asked to enter a password to encrypt/decrypt the keyring, or you may be - asked by your OS to confirm whether to give the DANDI CLI access to the + asked by your operating system to confirm whether to give the DANDI CLI access to the keyring. From b6078388a3fdff0859dcff7e8cf6fc966d592480 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Wed, 6 Mar 2024 20:25:25 -0500 Subject: [PATCH 47/53] State explicitly that our current maximal length of the path in dandiset must not exceed 512 ATM, if my check is right -- maximal length we can encounter is only 192, so we are quite ok dandi@drogon:/mnt/backup/dandi/dandisets$ pwd /mnt/backup/dandi/dandisets dandi@drogon:/mnt/backup/dandi/dandisets$ declare m=0;for ds in 00*; do git -C $ds ls-tree -r --name-only HEAD; done | awk '{ if (length($0) > max) max = length($0) } END { print max }' 192 When user manages to get over -- server 500s. Filed https://github.com/dandi/dandi-archive/issues/1889 --- docs/about/policies.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/about/policies.md b/docs/about/policies.md index bf072b2c..3c0d0b43 100644 --- a/docs/about/policies.md +++ b/docs/about/policies.md @@ -19,6 +19,7 @@ [Neuroimaging Data Model](NIDM), and other [BRAIN Initiative](https://braininitiative.nih.gov/) standards. We are working with the community to improve these standards and to make DANDI archive FAIR. +- **File name/path limitations:** There is a limit of 512 characters for the full path length within a dandiset. - **Volume and size limitations:** There is a limit of 5TB per file. We currently accept any size of standardized datasets, as long as you can upload them over an HTTPS connection. However, we ask you contact us if you plan to upload more than 10TB of data. From 761e460f1f80090e71c3d786c3302c1065ea9b1e Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Thu, 7 Mar 2024 13:21:05 -0500 Subject: [PATCH 48/53] Move tech limitations to "Using DANDI" section from the policy --- docs/10_using_dandi.md | 7 +++++++ docs/about/policies.md | 6 +----- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/10_using_dandi.md b/docs/10_using_dandi.md index 7e259a46..2cc86897 100644 --- a/docs/10_using_dandi.md +++ b/docs/10_using_dandi.md @@ -103,6 +103,13 @@ To use the hub, you will need to register for an account using the [DANDI Web application](https://dandiarchive.org/). Note that `Dandihub` is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data. +## Technical limitations + +- **File name/path:** There is a limit of 512 characters for the full path length within a dandiset. +- **Volume and size:** There is a limit of 5TB per file. We currently + accept any size of standardized datasets, as long as you can upload them over + an HTTPS connection. However, we ask you contact us if you plan to upload more than 10TB of data. + ## Citing DANDI You can add the following statement to the methods section of your manuscript. diff --git a/docs/about/policies.md b/docs/about/policies.md index 3c0d0b43..72ef1ff4 100644 --- a/docs/about/policies.md +++ b/docs/about/policies.md @@ -1,4 +1,4 @@ -# General Policies v1.0.1 +# General Policies v1.1.0 ## Content @@ -19,10 +19,6 @@ [Neuroimaging Data Model](NIDM), and other [BRAIN Initiative](https://braininitiative.nih.gov/) standards. We are working with the community to improve these standards and to make DANDI archive FAIR. -- **File name/path limitations:** There is a limit of 512 characters for the full path length within a dandiset. -- **Volume and size limitations:** There is a limit of 5TB per file. We currently - accept any size of standardized datasets, as long as you can upload them over - an HTTPS connection. However, we ask you contact us if you plan to upload more than 10TB of data. - **Data quality:** All data are provided “as-is”, and the user shall hold DANDI and data providers supplying data to the DANDI Archive free and harmless in connection with the use of such data. From 352ba93aa497c0919aa72c301356926a234a4d78 Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 26 Mar 2024 15:03:50 -0400 Subject: [PATCH 49/53] Add information on SEntry and heroku logs --- docs/40_development.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/40_development.md b/docs/40_development.md index ef6026d9..c7908df6 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -86,6 +86,21 @@ controls Netlify settings. The @dandibot GitHub account is the "owner" of the Netlify account used for this purpose; in order to get access to that account, speak to @satra. +## Logging + +### Sentry + +Sentry is used for error tracking main deployment. +To access Sentry, login to https://dandiarchive.sentry.io . + +### Heroku & Papertrail + +`dandi-api` and `dandi-api-staging` apps have Papertrail add-on configured to capture logs. +To access papertrail login to [Heroku dashboard](https://dashboard.heroku.com), proceed to the corresponding app and click on "Papertrail" add-on. + +CRON job on `drogon` server backups Papertrail logs as .csv files hourly at `/mnt/backup/dandi/papertrail-logs/{app}`. +Moreover, `heroku logs` processes per app dump logs to `/mnt/backup/dandi/heroku-logs/{app}` directory. + ## Code Hosting All code repositories are hosted on GitHub. The easiest way to contribute is to From 12e3097ba6933e167d9317a4df25ac534914501c Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Tue, 26 Mar 2024 16:43:35 -0400 Subject: [PATCH 50/53] Add on CI jobs logging and monitoring (upptime) --- docs/40_development.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/40_development.md b/docs/40_development.md index c7908df6..ce73303f 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -86,6 +86,15 @@ controls Netlify settings. The @dandibot GitHub account is the "owner" of the Netlify account used for this purpose; in order to get access to that account, speak to @satra. +## Monitoring + +### Service(s) status + +The DANDI project uses [upptime](https://upptime.js.org/) to monitor the status of DANDI provided and third-party services. +The configuration is available in [.upptimerc.yml](https://github.com/dandi/upptime/blob/master/.upptimerc.yml) of the https://github.com/dandi/upptime repository, which is automatically updated by the upptime project pipelines. +Upptime automatically opens new issues if any service becomes unresponsive, and closes issues whenever service comes back online. +https://www.dandiarchive.org/upptime/ is the public dashboard for the status of DANDI services. + ## Logging ### Sentry @@ -101,6 +110,11 @@ To access papertrail login to [Heroku dashboard](https://dashboard.heroku.com), CRON job on `drogon` server backups Papertrail logs as .csv files hourly at `/mnt/backup/dandi/papertrail-logs/{app}`. Moreover, `heroku logs` processes per app dump logs to `/mnt/backup/dandi/heroku-logs/{app}` directory. +### Continuous Integration (CI) Jobs + +The DANDI project uses GitHub Actions for continuous integration. +Logs for many of the repositories are archived on `drogon` server at `/mnt/backup/dandi/tinuous-logs/`. + ## Code Hosting All code repositories are hosted on GitHub. The easiest way to contribute is to From 88a971a7127a45435446ffd42c103b063a8e28b0 Mon Sep 17 00:00:00 2001 From: Roni Choudhury <2903332+waxlamp@users.noreply.github.com> Date: Wed, 27 Mar 2024 11:33:09 -0400 Subject: [PATCH 51/53] Apply a few grammar fixes --- docs/40_development.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/40_development.md b/docs/40_development.md index ce73303f..8c9021f4 100644 --- a/docs/40_development.md +++ b/docs/40_development.md @@ -104,10 +104,10 @@ To access Sentry, login to https://dandiarchive.sentry.io . ### Heroku & Papertrail -`dandi-api` and `dandi-api-staging` apps have Papertrail add-on configured to capture logs. -To access papertrail login to [Heroku dashboard](https://dashboard.heroku.com), proceed to the corresponding app and click on "Papertrail" add-on. +The `dandi-api` and `dandi-api-staging` apps have the Papertrail add-on configured to capture logs. +To access Papertrail, log in to the [Heroku dashboard](https://dashboard.heroku.com), proceed to the corresponding app and click on the "Papertrail" add-on. -CRON job on `drogon` server backups Papertrail logs as .csv files hourly at `/mnt/backup/dandi/papertrail-logs/{app}`. +A cronjob on the `drogon` server backs up Papertrail logs as .csv files hourly at `/mnt/backup/dandi/papertrail-logs/{app}`. Moreover, `heroku logs` processes per app dump logs to `/mnt/backup/dandi/heroku-logs/{app}` directory. ### Continuous Integration (CI) Jobs From 8eb60028a6906a647774a671e2a21820b249e235 Mon Sep 17 00:00:00 2001 From: Austin Macdonald Date: Wed, 27 Mar 2024 16:42:41 -0400 Subject: [PATCH 52/53] Document the possibility of extra dandihub images in the user docs --- docs/50_hub.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index b817ec94..9eb11b95 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -19,6 +19,12 @@ We request that users of this server be considerate of their usage of the DANDI Training large deep neural networks is not appropriate. A "Base (MATLAB)" server is also available, which provides a MATLAB cloud installation but you would be required to provide your own license. +## Custom server image + +If you need additional software installed in the image, you can add a server image option by making +a Pull Request to the [jupyterhub config file](https://github.com/dandi/dandi-hub/blob/do-eks/helm/jupyterhub/dandihub.yaml) as explained in +the [Dandihub readme](https://github.com/dandi/dandi-hub/tree/do-eks?tab=readme-ov-file#adjusting-available-server-options) + ## Example notebooks The best way to share analyses on DANDI data is through the DANDI example notebooks. @@ -26,4 +32,4 @@ These notebooks are maintained in the [dandi/example-notebooks](https://github.c Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications. Notebooks can be added and updated through a pull request to the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository. -Once the pull request is merged, your contributed notebook will be available to all DANDI Hub users. \ No newline at end of file +Once the pull request is merged, your contributed notebook will be available to all DANDI Hub users. From d662f4710ae054ec78a3268ca43654231d8736bf Mon Sep 17 00:00:00 2001 From: Austin Macdonald Date: Wed, 27 Mar 2024 17:57:32 -0400 Subject: [PATCH 53/53] accept Kabi suggestion Co-authored-by: Kabilar Gunalan --- docs/50_hub.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/50_hub.md b/docs/50_hub.md index 9eb11b95..a33fc139 100644 --- a/docs/50_hub.md +++ b/docs/50_hub.md @@ -21,9 +21,7 @@ A "Base (MATLAB)" server is also available, which provides a MATLAB cloud instal ## Custom server image -If you need additional software installed in the image, you can add a server image option by making -a Pull Request to the [jupyterhub config file](https://github.com/dandi/dandi-hub/blob/do-eks/helm/jupyterhub/dandihub.yaml) as explained in -the [Dandihub readme](https://github.com/dandi/dandi-hub/tree/do-eks?tab=readme-ov-file#adjusting-available-server-options) +If you need additional software installed in the image, you can add a server image that will be made available for all users in the `Server Options` menu. Add a server image by updating the `profileList` in the [JupyterHub config file](https://github.com/dandi/dandi-hub/blob/do-eks/helm/jupyterhub/dandihub.yaml) and submitting a pull request to the [dandi-hub repository](https://github.com/dandi/dandi-hub). Once the pull request is merged, the DANDI team will redeploy JupyterHub and the image will be available. ## Example notebooks