diff --git a/docs/source/conf.py b/docs/source/conf.py index 8f3d2b4..d6a19f0 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -30,6 +30,7 @@ 'sphinx.ext.extlinks', 'sphinx_design', 'sphinx_rtd_theme', + 'sphinx_tabs.tabs', ] intersphinx_disabled_domains = ['std'] diff --git a/docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst b/docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst index 35e074d..7126117 100644 --- a/docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst +++ b/docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst @@ -1,3 +1,5 @@ +.. _extension-create: + Creating an extension ===================== diff --git a/docs/source/extensions_tutorial/3_spec_api.rst b/docs/source/extensions_tutorial/3_spec_api.rst index d06f390..9969977 100644 --- a/docs/source/extensions_tutorial/3_spec_api.rst +++ b/docs/source/extensions_tutorial/3_spec_api.rst @@ -1,3 +1,5 @@ +.. _extension-spec-api: + The Spec API ------------ diff --git a/docs/source/extensions_tutorial/4_auto_api.rst b/docs/source/extensions_tutorial/4_auto_api.rst index 6504493..8fac63c 100644 --- a/docs/source/extensions_tutorial/4_auto_api.rst +++ b/docs/source/extensions_tutorial/4_auto_api.rst @@ -1,5 +1,9 @@ +.. _extension-auto-api: + Generating an API for an extension ------------------------------------------------------ +------------------------------------ + +.. _extension-auto-matlabnapi: Generating a MatNWB API ~~~~~~~~~~~~~~~~~~~~~~~ @@ -13,6 +17,8 @@ In MatNWB, simply call ``generateExtension("path/to/extension/namespace.yaml");` .. note:: As seen above, MatNWB will convert namespace names if they are not valid identifiers in MATLAB. See `Variable Names `_ for more information. In most cases, the conversion conforms with MATLAB's approach with `matlab.lang.makeValidName() `_ +.. _extension-auto-pythonapi: + Generating a PyNWB API ~~~~~~~~~~~~~~~~~~~~~~ @@ -36,7 +42,7 @@ created when you use the ``load_namespaces`` flag on :py:func:`~pynwb.NWBHDF5IO. Customizing automatically generated APIs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Once these classes are generated, you can customize them by dynamically adding or replacing attributes/methods (a.k.a., monkey patching). +Once these classes are generated, you can customize them by dynamically adding or replacing attributes/methods (a.k.a., monkey patching). A typical example is adding methods. Let's say you wanted a method that could return data from only the first channel. You could add that method like this: @@ -72,7 +78,7 @@ next section. .. tip:: This approach is easy, but note your API will be locked to your specification. If you make changes to your specification there will be corresponding changes to the API, and this is likely to break existing code. - Also, monkey patches can be very confusing to someone who is not aware of them. Differences - between the installed module and the actual behavior of the source code can lead to frustrated + Also, monkey patches can be very confusing to someone who is not aware of them. Differences + between the installed module and the actual behavior of the source code can lead to frustrated developers. As such, this approach should be used with great care. In the next section we will show you how to create your own custom API that is more robust. diff --git a/docs/source/extensions_tutorial/5_custom_api.rst b/docs/source/extensions_tutorial/5_custom_api.rst index 29c839d..e6283ce 100644 --- a/docs/source/extensions_tutorial/5_custom_api.rst +++ b/docs/source/extensions_tutorial/5_custom_api.rst @@ -1,3 +1,5 @@ +.. _extension-custom-api: + Building a custom Python API for an extension ============================================= diff --git a/docs/source/extensions_tutorial/6_documenting_extension.rst b/docs/source/extensions_tutorial/6_documenting_extension.rst index 0286f85..52bd847 100644 --- a/docs/source/extensions_tutorial/6_documenting_extension.rst +++ b/docs/source/extensions_tutorial/6_documenting_extension.rst @@ -1,3 +1,5 @@ +.. _extension-documentation: + Documenting Extensions ---------------------- diff --git a/docs/source/extensions_tutorial/7_publishing_extensions.rst b/docs/source/extensions_tutorial/7_publishing_extensions.rst index 5edbaef..033dff2 100644 --- a/docs/source/extensions_tutorial/7_publishing_extensions.rst +++ b/docs/source/extensions_tutorial/7_publishing_extensions.rst @@ -1,3 +1,5 @@ +.. _extension-publishing: + Publishing extensions --------------------- @@ -11,10 +13,12 @@ As illustrated in the figure, the publication process is divided into three main releases via PyPI, and 3) open publication of the extension to the community via the :ndx-catalog:`NDX Catalog <>`. +.. _extension-publishing-github: + Open Source: Releasing your extension Git repository ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -1. Before publishing your licence online you should add a license file. Permissive licenses should be used if possible. A `BSD license ` is recommended. +1. Before publishing your extension online you should add a license file. Permissive licenses should be used if possible. A `BSD license ` is recommended. 2. Modify ``README.md`` at the root directory of your extension repo to describe the extension for interested developers and users. 3. The first step to publishing your extension then is to make your Git repository accessible online via GitHub, or any other public Git hosting service of your choice. To publish your extension @@ -32,6 +36,8 @@ Open Source: Releasing your extension Git repository extensions repositories. However, users may chose to use other services (e.g., GitLab or Bitbucket) to share their sources. +.. _extension-publishing-pypi: + Open Access: Releasing your extension on PyPI ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -48,6 +54,8 @@ To make your extension installable for users via pip and manage public releases python setup.py sdist bdist_wheel twine upload dist/* +.. _extension-publishing-ndxcatalog: + Open Publication: Publishing your extension on the NDX Catalog ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/docs/source/extensions_tutorial/8_examples.rst b/docs/source/extensions_tutorial/8_examples.rst new file mode 100644 index 0000000..07e2ee6 --- /dev/null +++ b/docs/source/extensions_tutorial/8_examples.rst @@ -0,0 +1,10 @@ +.. _extension-nwb-examples: + +Examples +-------- + +.. toctree:: + :maxdepth: 1 + :titlesonly: + + extension_examples/labmetadata_extension diff --git a/docs/source/extensions_tutorial/extension_examples/labmetadata_extension.rst b/docs/source/extensions_tutorial/extension_examples/labmetadata_extension.rst new file mode 100644 index 0000000..abd11c0 --- /dev/null +++ b/docs/source/extensions_tutorial/extension_examples/labmetadata_extension.rst @@ -0,0 +1,346 @@ +.. _extension-example-labmetadata: + +Extensions for lab-specific metadata: Extending ``LabMetaData`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Use case +"""""""" + +.. short_description_start + +Here we address the use case of adding lab-specific metadata to a file, e.g., +lab-specific information about experimental protocols, lab-specific identifiers and so on. +This approach is intended for usually small metadata. :bdg-link-primary:`Extension source ` + +.. short_description_end + + + +Approach +"""""""" + +To include lab-specific metadata, NWB provides :py:class:`pynwb.file.LabMetaData` as a +a convenient base type, which makes it easy to add your data to an :py:class:`pynwb.file.NWBFile` +without having to modify the :py:class:`pynwb.file.NWBFile` type itself +(since adding of :py:class:`pynwb.file.LabMetaData` is already implemented). + +.. note:: + + NWB uses dynamically extensible table structures based on :py:class:`~hdmf.common.table.DynamicTable` + to describe metadata and derived results, e.g., :py:class:`~pynwb.epochs.TimeIntervals` for epochs or trials + or :py:class:`~pynwb.file.ElectrodeTable` to describe extracellular electrodes. Depending on the + type of metadata, use of these existing dynamic table structures can help avoid the need for + custom extensions by including the data as additional, custom columns in the appropriate existing tables. + +Creating the extension +"""""""""""""""""""""" + +**1.** Create a new repository for the extension using the :nwb_extension_git:`ndx-template`: + +.. code-block:: bash + + cookiecutter gh:nwb-extensions/ndx-template + +**2.** Answer a few simple questions of the cookiecutter template. We can respond to many questions +with ``Enter`` to accept the default response (e.g., to start with ``version=0.1.0``): + +.. code-block:: none + + namespace [ndx-my-namespace]: ndx-labmetadata-example + description [My NWB extension]: Example extension to illustrate how to extend LabMetaData for adding lab-specific metadata + author [My Name]: Oliver Ruebel + email [my_email@example.com]: oruebel@lbl.gov + github_username [myname]: oruebel + copyright [2021, Oliver Ruebel]: + version [0.1.0]: + release [alpha]: + Select license: + 1 - BSD-3 + 2 - MIT + 3 - Apache Software License 2.0 + 4 - Other + Choose from 1, 2, 3, 4 [1]: 1 + py_pkg_name [ndx_labmetadata_example]: + +**3.** Edit ``ndx-my-brainlabsrc/spec/create_extension_spec.py`` that was generated for you to define the +schema of your extension. See :ref:`extension-spec-api` section for details on how to use the specification API. + +* Add ``LabMetaData`` as an include type + +.. code-block:: python + + ns_builder.include_type('LabMetaData', namespace='core') + +* Define your new ``LabMetaData`` type for your lab + +.. code-block:: python + + labmetadata_ext = NWBGroupSpec( + name='custom_lab_metadata', + doc='Example extension type for storing lab metadata', + neurodata_type_def='LabMetaDataExtensionExample', + neurodata_type_inc='LabMetaData', + ) + +* Add the ``Groups``, ``Datasets``, and ``Attributes`` with the metadata specific to our lab to + our ``LabMetaData`` schema + +.. code-block:: python + + labmetadata_ext.add_dataset( + name="tissue_preparation", + doc="Lab-specific description of the preparation of the tissue", + dtype='text', + quantity='?' + ) + +* Add our new type definitions to the extension + +.. code-block:: python + + new_data_types = [labmetadata_ext] + +**4.** Generate the schema for the extension by running the ``create_extension_spec.py`` script + +.. code-block:: bash + + cd ndx-labmetadata-example + python src/spec/create_extension_spec.py + +**5.** Edit ``src/pynwb/__init__.py`` to define Python API classes for our new extension data types via :py:meth:`pynwb.get_class`. + +.. code-block:: python + + LabMetaDataExtensionExample = get_class('LabMetaDataExtensionExample', 'ndx-labmetadata-example') + +**6.** Define unit tests for the extension. The :nwb_extension_git:`ndx-template` created an example test +module ``src/pynwb/tests/test_tetrodeseries.py`` to illustrate how to implement tests. Here we simply remove +this file and replace it with our own tests `test_labmetadata_example.py `_. More details below in :ref:`extension-example-labmetadata-unittest`. + +**7.** To make sure our extension schema and source code files are version controlled, we now add all the files we just created to the Git repo: + +.. code-block:: bash + + git add . + git commit -m "Added API classes, tests, and schema files" + + +**8.** Install your extension (Python only)(Optional) + +.. code-block:: bash + + pip install . + +Now our extension is ready to use! + + +Creating custom Python API classes +"""""""""""""""""""""""""""""""""" + +We skip this step here, since this extension of :py:class:`~pynwb.file.LabMetaData` is simple enough that the +autogenerated class is sufficient. If the autogenerated class from :py:meth:`pynwb.get_class` for an extension +data types is not sufficient, then we can either customize the autogenerated class as described in +:ref:`extension-auto-pythonapi` (recommended only for basic changes) or define our own custom API class as +described in :ref:`extension-custom-api` (recommended for full customization). + + +.. _extension-example-labmetadata-unittest: + +Creating unit tests +""""""""""""""""""" + +.. tabs:: + + .. tab:: Python + + .. tabs:: + + .. code-tab:: py Unit test + + from pynwb.testing.mock.file import mock_NWBFile + from pynwb.testing import TestCase + from ndx_labmetadata_example import LabMetaDataExtensionExample + + + class TestLabMetaDataExtensionExample(TestCase): + """Test basic functionality of LabMetaDataExtensionExample without read/write""" + + def setUp(self): + """Set up an NWB file. Necessary because TetrodeSeries requires references to electrodes.""" + self.nwbfile = mock_NWBFile() + + def test_constructor(self): + """Test that the constructor for TetrodeSeries sets values as expected.""" + tissue_preparation = "Example tissue preparation" + lmdee_object = LabMetaDataExtensionExample(tissue_preparation=tissue_preparation) + self.assertEqual(lmdee_object.tissue_preparation, tissue_preparation) + + .. code-tab:: py Roundtrip test (read/write) + + from pynwb.testing.mock.file import mock_NWBFile + from pynwb.testing import TestCase + from pynwb.testing.testh5io import NWBH5IOMixin + from ndx_labmetadata_example import LabMetaDataExtensionExample + + class TestLabMetaDataExtensionExampleRoundtrip(NWBH5IOMixin, TestCase): + """ + Roundtrip test for LabMetaDataExtensionExample to test read/write + + This test class writes the LabMetaDataExtensionExample to an NWBFile, then + reads the data back from the file, and compares that the data read from file + is consistent with the original data. Using the pynwb.testing infrastructure + simplifies this complex test greatly by allowing to simply define how to + create the container, add to a file, and retrieve it form a file. The + task of writing, reading, and comparing the data is then taken care of + automatically by the NWBH5IOMixin. + """ + + def setUpContainer(self): + """set up example LabMetaDataExtensionExample object""" + self.lab_meta_data = LabMetaDataExtensionExample(tissue_preparation="Example tissue preparation") + return self.lab_meta_data + + def addContainer(self, nwbfile): + """Add the test LabMetaDataExtensionExample to the given NWBFile.""" + nwbfile.add_lab_meta_data(lab_meta_data=self.lab_meta_data) + + def getContainer(self, nwbfile): + """Get the LabMetaDataExtensionExample object from the given NWBFile.""" + return nwbfile.get_lab_meta_data(self.lab_meta_data.name) + + + .. code-tab:: bash Running Python unit tests + + cd ndx-labmetadata-example + pytest + + .. tab:: MATLAB + + .. tabs:: + + .. code-tab:: c Unit test + + Coming soon ... + + .. code-tab:: c Roundtrip test (read/write) + + Coming soon ... + + .. code-tab:: bash Running MATLAB unit tests + + Coming soon ... + + +Documenting the extension +""""""""""""""""""""""""" + +* **REAME.md:** Add instructions to the ``README.md`` file. This typically includes information on how to install the + extension and an example on how to use the extension +* **Schema and user documentation:** + + * Install the latest release of hdmf_docutils: ``python -m pip install hdmf-docutils`` + * Generate the documentation for your extension based on the YAML schema files via: + + .. code-block:: bash + + cd docs/ + make html + + * To view the docs, simply open ``docs/build/html/index.html`` in your browser + * See the `docs/README.md `_ + for instructions on how to customize the documentation for your extension. + +See :ref:`extension-documentation` for more details. + +Writing data using the extension +"""""""""""""""""""""""""""""""" + +.. tabs:: + + .. code-tab:: py Python + + from pynwb.file import NWBFile, Subject + from ndx_labmetadata_example import LabMetaDataExtensionExample + from pynwb import NWBHDF5IO + from uuid import uuid4 + from datetime import datetime + + # create an example NWBFile + nwbfile = NWBFile( + session_description="test session description", + identifier=str(uuid4()), + session_start_time=datetime(1970, 1, 1), + subject=Subject( + age="P50D", + description="example mouse", + sex="F", + subject_id="test_id") + ) + + # create our custom lab metadata + lab_meta_data = LabMetaDataExtensionExample(tissue_preparation="Example tissue preparation") + + # Add the test LabMetaDataExtensionExample to the NWBFile + nwbfile.add_lab_meta_data(lab_meta_data=lab_meta_data) + + # Write the file to disk + filename = "testfile.nwb" + with NWBHDF5IO(path=filename, mode="a") as io: + io.write(nwbfile) + + .. code-tab:: c MATLAB + + Coming soon ... + + +Reading an NWB file that uses the extension +""""""""""""""""""""""""""""""""""""""""""" + +.. tabs:: + + .. code-tab:: py Python + + from pynwb import NWBHDF5IO + from ndx_labmetadata_example import LabMetaDataExtensionExample + + # Read the file from disk + io = NWBHDF5IO(path=filename, mode="r") + nwbfile = io.read() + # Get the custom lab metadata object + lab_meta_data = nwbfile.get_lab_meta_data(name="custom_lab_metadata") + + .. code-tab:: py Python (without extension installed) + + from pynwb import NWBHDF5IO + + # Read the file from disk. Load the namespace from file to + # autogenerate classes from the schema + io = NWBHDF5IO(path=filename, mode="r", load_namespaces=True) + nwbfile = io.read() + # Get the custom lab metadata object + lab_meta_data = nwbfile.get_lab_meta_data(name="custom_lab_metadata") + + .. code-tab:: c MATLAB + + Coming soon ... + + + +Publishing the extension +""""""""""""""""""""""""" + +The steps to publish an extension are the same for all extensions. We, therefore, here only briefly describe +he main steps for publishing our extension. For a more in-depth guide, see the page :ref:`extension-publishing` + +* **GitHub (Open Source):** To make the sources of your extension openly accessible, publish the extension + on GitHub by following the instructions on :ref:`extension-publishing-github`. + +* **PyPI (Open Access):** Publish your extension on `PyPI `_ to make it easy for users to + install it and to create a persistent release of the extension following the :ref:`extension-publishing-pypi` guide. + +* **NDX Catalog (Open Publication)**: The :ndx-catalog:`NDX Catalog <>` serves as a central, community-led catalog + for extensions to the NWB data standard. The NDX Catalog manages basic metadata about extensions while ownership + of the source repositories for the extensions remain with the developers. For a step-by-step guide the + :ref:`extension-publishing-ndxcatalog` guide. + + diff --git a/docs/source/extensions_tutorial/extensions_tutorial_home.rst b/docs/source/extensions_tutorial/extensions_tutorial_home.rst index 387cf90..027d9f8 100644 --- a/docs/source/extensions_tutorial/extensions_tutorial_home.rst +++ b/docs/source/extensions_tutorial/extensions_tutorial_home.rst @@ -18,3 +18,4 @@ the `NWB Specification Language =1 # <1 does not work with docutils>=0.17 sphinx_design +sphinx-tabs