merge main into ak-dandi-init

.github/workflows/build-docs.yaml

@@ -0,0 +1,22 @@
+name: Build MkDocs
+on:
+  push:
+  pull_request:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v1
+
+      - name: Install version of Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.11' 
+
+      - name: Install requirements
+        run: pip install -r requirements.txt
+
+      - name: Build docs
+        run: mkdocs build

README.md

@@ -1,5 +1,5 @@
 # DANDI Handbook
-Handbook for interacting with the DANDI archive.
+Handbook for interacting with the DANDI Archive.
 
 ## DANDI Style Guidelines
 
@@ -16,6 +16,7 @@ Follow the guidelines below when creating and revising text in the DANDI Handboo
 *	**license** (**not** licence); in general, prefer American spelling
 *	limited use of "please"
 *	steps should start with 1, not 0
+*	**DANDI Archive** - capitalize "archive" if it follows DANDI (**not** DANDI archive)
 
 ## HOWTO
 

docs/10_using_dandi.md

@@ -103,17 +103,24 @@ To use the hub, you will need to register for an
 account using the [DANDI Web application](https://dandiarchive.org/). 
 Note that `Dandihub` is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data.
 
+## Technical limitations
+
+- **File name/path:** There is a limit of 512 characters for the full path length within a dandiset.
+- **Volume and size:** There is a limit of 5TB per file. We currently
+  accept any size of standardized datasets, as long as you can upload them over
+  an HTTPS connection. However, we ask you contact us if you plan to upload more than 10TB of data.
+
 ## Citing DANDI
 
 You can add the following statement to the methods section of your manuscript.
 
 > Data and associated metadata were uploaded to the DANDI archive [RRID:SCR_017571] using 
-  the Python command line tool (https://doi.org/10.5281/zenodo.7041535). The data were first 
+  the Python command line tool (https://doi.org/10.5281/zenodo.3692138). The data were first 
   converted into the NWB format (https://doi.org/10.1101/2021.03.13.435173) and  organized 
   into a BIDS-like (https://doi.org/10.1038/sdata.2016.44) structure.
 
 You can refer to DANDI using any of the following options:
 
 * Using an RRID [RRID:SCR_017571](https://scicrunch.org/scicrunch/Resources/record/nlx_144509-1/SCR_017571/resolver). 
 
-* Using the DANDI CLI reference: https://doi.org/10.5281/zenodo.7041535
+* Using the DANDI CLI reference: https://doi.org/10.5281/zenodo.3692138

docs/13_upload.md

@@ -2,25 +2,14 @@
 
 To create a new Dandiset and upload your data, you need to have a DANDI account.
 
-## Create an Account on DANDI
-
-To create a DANDI account:
-
-1. [Create a GitHub account](https://github.com/) if you don't have one.
-2. Using your GitHub account, [register a DANDI account](https://gui.dandiarchive.org/#/user/register).
-
-You will receive an email acknowledging activation of your account within 24
-hours, after which you can log in to DANDI using GitHub by clicking the login
-button.
-
 ## Create a Dandiset and Add Data
 
-You can create a new Dandiset at [http://dandiarchive.org](http://dandiarchive.org). This Dandiset can be fully 
+You can create a new Dandiset at https://dandiarchive.org. This Dandiset can be fully 
 public or embargoed 
 according to NIH policy.
 When you create a Dandiset, a permanent ID is automatically assigned to it.
 To prevent the production server from being inundated with test Dandisets, we encourage developers to develop 
-against the development server ([https://gui-staging.dandiarchive.org/](https://gui-staging.dandiarchive.org/)). Note 
+against the development server (https://gui-staging.dandiarchive.org/). Note 
 that the development server
 should not be used to stage your data. All data are uploaded as draft and can be adjusted before publishing on
 the production server. The development server is primarily used by users learning to use DANDI or by developers.
@@ -30,6 +19,7 @@ two different servers differ slightly.
 
 ### **Setup**
 
+1. To create a new Dandiset and upload your data, you need to have a DANDI account. See the [Create a DANDI Account](./16_account.md) page.
 1. Log in to DANDI and copy your API key. Click on your user initials in the
     top-right corner after logging in. Production (dandiarchive.org) and staging (gui-staging.dandiarchive.org) servers 
       have different API keys and different logins.
@@ -101,42 +91,47 @@ If you have an issue using the Python CLI, see the [Dandi Debugging section](./1
 
 ## Storing Access Credentials
 
-By default, the DANDI CLI looks for an API key in the `DANDI_API_KEY`
-environment variable.  To set this on Linux or macOS, run
-
-```bash
-export DANDI_API_KEY=personal-key-value
-```
-*Note that there are no spaces around the "=".
-
-If this is not set, the CLI will look up the API
-key using the [keyring](https://github.com/jaraco/keyring) library, which
-supports numerous backends, including the system keyring, an encrypted keyfile,
-and a plaintext (unencrypted) keyfile.
-
-- You can store your API key where the `keyring` library can find it by using
-  the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the
-  API key when asked for the password for `key` in `dandi-api-dandi`.
-
-- You can set the backend the `keyring` library uses either by setting the
-  `PYTHON_KEYRING_BACKEND` environment variable or by filling in [the `keyring`
-  library's configuration file](https://github.com/jaraco/keyring#configuring).
-  IDs for the available backends can be listed by running `keyring --list`.  If
-  no backend is specified in this way, the library will use the available
-  backend with the highest priority.
-
-If the API key isn't stored in either the `DANDI_API_KEY` environment variable
-or in the keyring, the CLI will prompt you to enter the API key, and then it
-will store it in the keyring.  This may cause you to be prompted further; you
-may be asked to enter a password to encrypt/decrypt the keyring, or you may be
-asked by your OS to confirm whether to give the DANDI CLI access to the
-keyring.
-
-- If the DANDI CLI encounters an error while attempting to fetch the API key
-  from the default keyring backend, it will fall back to using an encrypted
-  keyfile (the `keyrings.alt.file.EncryptedKeyring` backend).  If the keyfile
-  does not already exist, the CLI will ask you for confirmation; if you answer
-  "yes," the `keyring` configuration file (if it does not already exist; see
-  above) will be configured to use `EncryptedKeyring` as the default backend.
-  If you answer "no," the CLI will exit with an error, and you must store the
-  API key somewhere accessible to the CLI on your own.
+There are two options for storing your DANDI access credentials.
+
+1. `DANDI_API_KEY` Environment Variable
+
+    - By default, the DANDI CLI looks for an API key in the `DANDI_API_KEY`
+      environment variable.  To set this on Linux or macOS, run:
+
+            export DANDI_API_KEY=personal-key-value
+
+    - Note that there are no spaces around the "=".
+
+2. `keyring` Library
+    - If the `DANDI_API_KEY` environment variable is not set, the CLI will look up the API
+        key using the [keyring](https://github.com/jaraco/keyring) library, which
+        supports numerous backends, including the system keyring, an encrypted keyfile,
+        and a plaintext (unencrypted) keyfile.
+
+    - Specifying the `keyring` backend
+        - You can set the backend the `keyring` library uses either by setting the
+          `PYTHON_KEYRING_BACKEND` environment variable or by filling in the `keyring`
+          library's [configuration file](https://github.com/jaraco/keyring#configuring).
+        - IDs for the available backends can be listed by running `keyring --list`.
+        - If no backend is specified in this way, the library will use the available
+          backend with the highest priority.
+        - If the DANDI CLI encounters an error while attempting to fetch the API key
+          from the default keyring backend, it will fall back to using an encrypted
+          keyfile (the `keyrings.alt.file.EncryptedKeyring` backend).  If the keyfile
+          does not already exist, the CLI will ask you for confirmation; if you answer
+          "yes," the `keyring` configuration file (if it does not already exist; see
+          above) will be configured to use `EncryptedKeyring` as the default backend.
+          If you answer "no," the CLI will exit with an error, and you must store the
+          API key somewhere accessible to the CLI on your own.
+
+    - Storing the API key with `keyring`
+        1. You can store your API key where the `keyring` library can find it by using
+          the `keyring` program: Run `keyring set dandi-api-dandi key` and enter the
+          API key when asked for the password for `key` in `dandi-api-dandi`.
+
+        2. If the API key isn't stored in either the `DANDI_API_KEY` environment variable
+          or in the keyring, the CLI will prompt you to enter the API key, and then it
+          will store it in the keyring.  This may cause you to be prompted further; you
+          may be asked to enter a password to encrypt/decrypt the keyring, or you may be
+          asked by your operating system to confirm whether to give the DANDI CLI access to the
+          keyring.

docs/14_publish.md

@@ -1,15 +1,35 @@
 # Publishing Dandisets
 
-Once you create a Dandiset, DANDI will automatically create a `draft` version of the Dandiset that 
-can be 
-changed as many times as needed. 
-
-Prior to publishing, you should edit your metadata appropriately
-(e.g. people and funding contributors, protocol information, keywords, related resources in other 
-sources such as publications, code repositories, and other data).
-If you need to change the Dandiset or its metadata in the future, you will need to create another version of your 
-Dandiset. The content of the specific published version of the Dandiset cannot be changed.
-
-When you are ready to publish your data and to create a unique version of your 
-Dandiset, click the `Publish` button (on the right panel of the landing page). Note that the `Publish` 
-button is only active if all the metadata and asset errors are fixed.
+Once you create a Dandiset, DANDI will automatically create a `draft` version
+of the Dandiset that can be changed as many times as needed by editing the 
+metadata or uploading new files.
+
+When the draft version is ready, you can *publish* your Dandiset. This results
+in an immutable snapshot of your Dandiset with its own unique version number
+that others can cite. If you need to change the data or metadata, you can do
+so by continuing to modify the draft version and publishing a new version
+when you are ready.
+
+Follow these steps to publish your Dandiset:
+
+1. Edit the Dandiset metadata, aiming to fix all Dandiset metadata validation
+   errors, and include any other useful information. For example, you may want
+   to edit the following fields:
+    - People and funding contributors
+    - Protocol information
+    - Keywords
+    - Related resources such as publications and code repositories
+
+1. Fix all asset metadata errors by modifying the asset files to eliminate
+   the errors and re-uploading them.
+
+1. When all the Dandiset metadata and asset metadata errors are fixed, and the Dandiset is made public if it was initially embargoed, the
+   `Publish` button (on the right panel of the Dandiset landing page) will
+   be enabled and turn green. Click the button to publish your Dandiset.
+
+1. In the lower right section of the Dandiset landing page, you should see
+   the new, published version of your Dandiset listed. Click on that link
+   to view this version.
+
+**NOTE:** Dandisets with Zarr assets currently cannot be published. We are 
+actively working on enabling this feature.

docs/16_account.md

@@ -0,0 +1,13 @@
+# Create a DANDI Account
+
+A DANDI account is only required for specific features.
+Without a DANDI account, you can search, view, and download files.
+With a DANDI account, you can additionally create Dandisets and access the DANDI Hub to analyze existing data.
+
+## Instructions
+
+1. To create a DANDI account, first [create a GitHub account](https://github.com/) if you don't have one.
+1. Using your GitHub account, register for a DANDI account by selecting the `LOG IN WITH GITHUB` button on the [DANDI homepage](https://dandiarchive.org).
+1. You will receive an email acknowledging that your request for an account will be reviewed within 24 hours.
+  - **note**: Requests from new GitHub accounts and from emails that are not an `.edu` domain might take longer to review and are more likely to be rejected, especially if there are no plans described to upload data to the archive.
+1. If your request for an account is approved, you will be able to log in to DANDI using GitHub by clicking the `LOG IN WITH GITHUB` button.

docs/40_development.md

@@ -86,6 +86,35 @@ controls Netlify settings. The @dandibot GitHub account is the "owner" of the
 Netlify account used for this purpose; in order to get access to that account,
 speak to @satra.
 
+## Monitoring
+
+### Service(s) status
+
+The DANDI project uses [upptime](https://upptime.js.org/) to monitor the status of DANDI provided and third-party services.
+The configuration is available in [.upptimerc.yml](https://github.com/dandi/upptime/blob/master/.upptimerc.yml) of the https://github.com/dandi/upptime repository, which is automatically updated by the upptime project pipelines.
+Upptime automatically opens new issues if any service becomes unresponsive, and closes issues whenever service comes back online.
+https://www.dandiarchive.org/upptime/ is the public dashboard for the status of DANDI services.
+
+## Logging
+
+### Sentry
+
+Sentry is used for error tracking main deployment.
+To access Sentry, login to https://dandiarchive.sentry.io .
+
+### Heroku & Papertrail
+
+The `dandi-api` and `dandi-api-staging` apps have the Papertrail add-on configured to capture logs.
+To access Papertrail, log in to the [Heroku dashboard](https://dashboard.heroku.com), proceed to the corresponding app and click on the "Papertrail" add-on.
+
+A cronjob on the `drogon` server backs up Papertrail logs as .csv files hourly at `/mnt/backup/dandi/papertrail-logs/{app}`.
+Moreover, `heroku logs` processes per app dump logs to `/mnt/backup/dandi/heroku-logs/{app}` directory.
+
+### Continuous Integration (CI) Jobs
+
+The DANDI project uses GitHub Actions for continuous integration.
+Logs for many of the repositories are archived on `drogon` server at `/mnt/backup/dandi/tinuous-logs/`.
+
 ## Code Hosting
 
 All code repositories are hosted on GitHub. The easiest way to contribute is to

docs/50_hub.md

@@ -0,0 +1,33 @@
+# Using the DANDI Hub
+
+[DANDI Hub](http://hub.dandiarchive.org) is a [JupyterHub](https://jupyterhub.readthedocs.io) instance in the cloud to interact with the data stored in DANDI, and is free to use for exploratory analysis of data on DANDI.
+For instructions on how to navigate JupyterHub see this [YouTube tutorial](https://www.youtube.com/watch?v=5pf0_bpNbkw&t=09m20s).
+Note that DANDI Hub is not intended for significant computation, but provides a place to introspect Dandisets and to perform some analysis and visualization of data.
+
+## Registration
+
+To use the [DANDI Hub](http://hub.dandiarchive.org), you must first register for an account using the [DANDI website](http://dandiarchive.org).
+See the [Create a DANDI Account](./16_account.md) page.
+
+## Choosing a server option
+
+When you start up the DANDI Hub, you will be asked to select across a number of server options.
+For basic exploration, Tiny or Base would most likely be appropriate.
+The DANDI Hub also currently offers Medium and Large options, which have more available memory and compute power.
+The "T4 GPU inference" server comes with an associated T4 GPU, and is intended to be used for applications that require GPU for inference.
+We request that users of this server be considerate of their usage of the DANDI Hub as a free community resource.
+Training large deep neural networks is not appropriate.
+A "Base (MATLAB)" server is also available, which provides a MATLAB cloud installation but you would be required to provide your own license.
+
+## Custom server image
+
+If you need additional software installed in the image, you can add a server image that will be made available for all users in the `Server Options` menu.  Add a server image by updating the `profileList` in the [JupyterHub config file](https://github.com/dandi/dandi-hub/blob/do-eks/helm/jupyterhub/dandihub.yaml) and submitting a pull request to the [dandi-hub repository](https://github.com/dandi/dandi-hub).  Once the pull request is merged, the DANDI team will redeploy JupyterHub and the image will be available.
+
+## Example notebooks
+
+The best way to share analyses on DANDI data is through the DANDI example notebooks.
+These notebooks are maintained in the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository which provides more information about their organization.
+Dandiset contributors are encouraged to use these notebooks to demonstrate how to read, analyze, and visualize the data, and how to produce figures from associated scientific publications.
+
+Notebooks can be added and updated through a pull request to the [dandi/example-notebooks](https://github.com/dandi/example-notebooks) repository.
+Once the pull request is merged, your contributed notebook will be available to all DANDI Hub users.

docs/about/policies.md

@@ -1,4 +1,4 @@
-# General Policies v1.0.1
+# General Policies v1.1.0
 
 ## Content
 
@@ -19,9 +19,6 @@
   [Neuroimaging Data Model](NIDM), and other [BRAIN Initiative](https://braininitiative.nih.gov/)
   standards. We are working with the community to improve these standards and to
   make DANDI archive FAIR.
-- **Volume and size limitations:** There is a limit of 5TB per file. We currently
-  accept any size of standardized datasets, as long as you can upload them over
-  an HTTPS connection. However, we ask you contact us if you plan to upload more than 10TB of data.
 - **Data quality:** All data are provided “as-is”, and the user shall hold
   DANDI and data providers supplying data to the DANDI Archive free and harmless in
   connection with the use of such data.

docs/index.md

@@ -5,7 +5,7 @@ src="./img/dandi-banner.png"
 alt="dandi_banner"
 style="width: 75%; height: auto; display: block; margin-left: auto;  margin-right: auto;"/>
 
-The Web interface to the DANDI archive is located at [https://dandiarchive.org](https://dandiarchive.org).
+The Web interface to the DANDI archive is located at https://dandiarchive.org.
 This documentation explains how to interact with the archive.
 
 ## How to Use This Documentation
@@ -24,14 +24,14 @@ page in the Developer Guide section.
 
 You can communicate with the DANDI team in a variety of ways, depending on your needs:
 
-- You can ask questions, report bugs, or
-  request features [at our helpdesk](https://github.com/dandi/helpdesk/issues/new/choose).
-- For interacting with the global neuroscience community, post on [https://neurostars.org](https://neurostars.org)
-  and use the tag [dandi](https://neurostars.org/tag/dandi).
-- You can use the DANDI Slack workspace, which we will invite you to after approving your [registration on
-  DANDI using GitHub](https://dandiarchive.org/) (this registration is required to upload data or to use the DANDI
+- You can ask questions, report bugs, or 
+request features [at our helpdesk](https://github.com/dandi/helpdesk/issues/new/choose).
+- For interacting with the global neuroscience community, post on https://neurostars.org
+and use the tag [dandi](https://neurostars.org/tag/dandi).
+- You can use the DANDI Slack workspace, which we will invite you to after approving your [registration on 
+  DANDI using GitHub](https://dandiarchive.org/) (this registration is required to upload data or to use the DANDI 
   JupyterHub). See [here for details on how to
-  register](./13_upload.md).
+  [register](./13_upload.md).
 - Email us: [[email protected]](mailto: [email protected])
 
 ## Contributing and Feedback