From e71b3d2582682e463729dbb21d73815a308b19e2 Mon Sep 17 00:00:00 2001 From: Ryan Ly Date: Mon, 19 Apr 2021 20:47:30 -0700 Subject: [PATCH] Update copyright year, move schema language repo (#471) --- Legal.txt | 2 +- README.rst | 9 +- docs/language/Makefile | 166 --- docs/language/Readme.md | 23 - docs/language/make.bat | 202 ---- .../source/_static/theme_overrides.css | 13 - docs/language/source/conf.py | 257 ----- docs/language/source/credits.rst | 43 - docs/language/source/index.rst | 21 - .../specification_language_description.rst | 966 ------------------ .../specification_language_release_notes.rst | 189 ---- license.txt | 2 +- 12 files changed, 8 insertions(+), 1885 deletions(-) delete mode 100644 docs/language/Makefile delete mode 100644 docs/language/Readme.md delete mode 100644 docs/language/make.bat delete mode 100644 docs/language/source/_static/theme_overrides.css delete mode 100644 docs/language/source/conf.py delete mode 100644 docs/language/source/credits.rst delete mode 100644 docs/language/source/index.rst delete mode 100644 docs/language/source/specification_language_description.rst delete mode 100644 docs/language/source/specification_language_release_notes.rst diff --git a/Legal.txt b/Legal.txt index 7fd7d417..9d3176d4 100644 --- a/Legal.txt +++ b/Legal.txt @@ -1,4 +1,4 @@ -“nwb-schema” Copyright (c) 2017-2020, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. +“nwb-schema” Copyright (c) 2017-2021, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at IPO@lbl.gov. diff --git a/README.rst b/README.rst index 07be7a7b..18935c98 100644 --- a/README.rst +++ b/README.rst @@ -7,9 +7,12 @@ This repo contains: * The core NWB schema * The `Documentation for the NWB schema `_ (editable `here `_) -* The `NWB specification language `_ (editable `here `_) * The `HDF5 storage specification `_ (editable `here `_) +The NWB schema uses the [NWB specification language](http://schema-language.readthedocs.io/), +which defines formal structures for describing the organization of +complex data using basic concepts, e.g., Groups, Datasets, Attributes, and Links. + For more information: - Learn more about NWB and `nwb.org `_. @@ -20,7 +23,7 @@ For more information: License ======================== -“nwb-schema” Copyright (c) 2017-2020, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. +“nwb-schema” Copyright (c) 2017-2021, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: @@ -38,7 +41,7 @@ You are under no obligation whatsoever to provide any bug fixes, patches, or upg Copyright ======================== -“nwb-schema” Copyright (c) 2017-2020, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. +“nwb-schema” Copyright (c) 2017-2021, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at IPO@lbl.gov. diff --git a/docs/language/Makefile b/docs/language/Makefile deleted file mode 100644 index 459712e8..00000000 --- a/docs/language/Makefile +++ /dev/null @@ -1,166 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -SPHINXAPIDOC = sphinx-apidoc -PAPER = -BUILDDIR = _build -RSTDIR = source - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) $(RSTDIR) -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext - -help: - @echo "To update documentation sources from the format specification please use \`make apidoc'" - @echo "" - @echo "To build the documenation please use \`make ' where is one of" - @echo " fulldoc to rebuild the html and latexpdf documents all at once" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - @echo " clean to clean all documents built by Sphinx in _build" - -clean: - -rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/sample.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/sample.qhc" - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/sample" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/sample" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." - -fulldoc: - $(MAKE) clean - @echo - @echo "Rebuilding html, latexpdf" - $(MAKE) html - $(MAKE) latexpdf diff --git a/docs/language/Readme.md b/docs/language/Readme.md deleted file mode 100644 index 9d022242..00000000 --- a/docs/language/Readme.md +++ /dev/null @@ -1,23 +0,0 @@ -**Overview** - -The NWB:N specification documentation uses Sphinx [http://www.sphinx-doc.org/en/stable/index.html](http://www.sphinx-doc.org/en/stable/index.html) - -**Building the documentation** - -To build the documentation simply: - -```make ``` - -where `````` is, e.g., ```latexpdf```, ```html```, ```singlehtml``` or ```man```. For a complete list of supported doc-types see: - -```make help``` - -**Cleaning up** - -By calling: - -```make clean``` - - - -. diff --git a/docs/language/make.bat b/docs/language/make.bat deleted file mode 100644 index dc484365..00000000 --- a/docs/language/make.bat +++ /dev/null @@ -1,202 +0,0 @@ -@ECHO OFF - -REM Command file for Sphinx documentation - -if "%SPHINXBUILD%" == "" ( - set SPHINXBUILD=sphinx-build -) -set BUILDDIR=_build -set RSTDIR=source -set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% %RSTDIR% -set I18NSPHINXOPTS=%SPHINXOPTS% . -if NOT "%PAPER%" == "" ( - set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS% - set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS% -) - -if "%1" == "" goto help - -if "%1" == "help" ( - :help - echo.Please use `make ^` where ^ is one of - echo. fulldoc to rebuild the html and latex documents all at once - echo. html to make standalone HTML files - echo. dirhtml to make HTML files named index.html in directories - echo. singlehtml to make a single large HTML file - echo. pickle to make pickle files - echo. json to make JSON files - echo. htmlhelp to make HTML files and a HTML help project - echo. qthelp to make HTML files and a qthelp project - echo. devhelp to make HTML files and a Devhelp project - echo. epub to make an epub - echo. latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter - echo. text to make text files - echo. man to make manual pages - echo. texinfo to make Texinfo files - echo. gettext to make PO message catalogs - echo. changes to make an overview over all changed/added/deprecated items - echo. linkcheck to check all external links for integrity - echo. doctest to run all doctests embedded in the documentation if enabled - echo. clean to clean all documents built by Sphinx in _build - goto end -) - -if "%1" == "clean" ( - for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i - del /q /s %BUILDDIR%\* - goto end -) - -if "%1" == "html" ( - %SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/html. - goto end -) - -if "%1" == "dirhtml" ( - %SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml. - goto end -) - -if "%1" == "singlehtml" ( - %SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml. - goto end -) - -if "%1" == "pickle" ( - %SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the pickle files. - goto end -) - -if "%1" == "json" ( - %SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can process the JSON files. - goto end -) - -if "%1" == "htmlhelp" ( - %SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run HTML Help Workshop with the ^ -.hhp project file in %BUILDDIR%/htmlhelp. - goto end -) - -if "%1" == "qthelp" ( - %SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; now you can run "qcollectiongenerator" with the ^ -.qhcp project file in %BUILDDIR%/qthelp, like this: - echo.^> qcollectiongenerator %BUILDDIR%\qthelp\sample.qhcp - echo.To view the help file: - echo.^> assistant -collectionFile %BUILDDIR%\qthelp\sample.ghc - goto end -) - -if "%1" == "devhelp" ( - %SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. - goto end -) - -if "%1" == "epub" ( - %SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The epub file is in %BUILDDIR%/epub. - goto end -) - -if "%1" == "latex" ( - %SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex - if errorlevel 1 exit /b 1 - echo. - echo.Build finished; the LaTeX files are in %BUILDDIR%/latex. - goto end -) - -if "%1" == "text" ( - %SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The text files are in %BUILDDIR%/text. - goto end -) - -if "%1" == "man" ( - %SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The manual pages are in %BUILDDIR%/man. - goto end -) - -if "%1" == "texinfo" ( - %SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo. - goto end -) - -if "%1" == "gettext" ( - %SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale - if errorlevel 1 exit /b 1 - echo. - echo.Build finished. The message catalogs are in %BUILDDIR%/locale. - goto end -) - -if "%1" == "changes" ( - %SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes - if errorlevel 1 exit /b 1 - echo. - echo.The overview file is in %BUILDDIR%/changes. - goto end -) - -if "%1" == "linkcheck" ( - %SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck - if errorlevel 1 exit /b 1 - echo. - echo.Link check complete; look for any errors in the above output ^ -or in %BUILDDIR%/linkcheck/output.txt. - goto end -) - -if "%1" == "doctest" ( - %SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest - if errorlevel 1 exit /b 1 - echo. - echo.Testing of doctests in the sources finished, look at the ^ -results in %BUILDDIR%/doctest/output.txt. - goto end -) - -if "%1" == "fulldoc" ( - make clean - echo. - echo.Rebuilding html, latex - make html - make latex - goto end -) - -:end diff --git a/docs/language/source/_static/theme_overrides.css b/docs/language/source/_static/theme_overrides.css deleted file mode 100644 index 63ee6cc7..00000000 --- a/docs/language/source/_static/theme_overrides.css +++ /dev/null @@ -1,13 +0,0 @@ -/* override table width restrictions */ -@media screen and (min-width: 767px) { - - .wy-table-responsive table td { - /* !important prevents the common CSS stylesheets from overriding - this as on RTD they are loaded after this stylesheet */ - white-space: normal !important; - } - - .wy-table-responsive { - overflow: visible !important; - } -} diff --git a/docs/language/source/conf.py b/docs/language/source/conf.py deleted file mode 100644 index e33350c4..00000000 --- a/docs/language/source/conf.py +++ /dev/null @@ -1,257 +0,0 @@ -# -*- coding: utf-8 -*- -# -# sample documentation build configuration file, created by -# sphinx-quickstart on Mon Apr 16 21:22:43 2012. -# -# This file is execfile()d with the current directory set to its containing dir. -# -# Not that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys, os -import sphinx_rtd_theme - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -#sys.path.insert(0, os.path.abspath('.')) - -autoclass_content = 'both' -autodoc_docstring_signature = True -autodoc_member_order = 'bysource' -add_function_parentheses = False -numfig = True - -# -- General configuration ----------------------------------------------------- - -# If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be extensions -# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.autodoc'] - -# Add any paths that contain templates here, relative to this directory. -templates_path = ['_templates'] - -# The suffix of source filenames. -source_suffix = '.rst' - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'source/index' -master_doc = 'index' - -# General information about the project. -project = u'NWB Specification Language' -copyright = u'2017, Neurodata Without Borders' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -version = 'v2.0.0' -# The full version, including alpha/beta/rc tags. -release = 'v2.0.0-beta' - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -#language = None - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['_build', 'test.py'] - -# The reST default role (used for this markup: `text`) to use for all documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# A list of ignored prefixes for module index sorting. -#modindex_common_prefix = [] - - -# -- Options for HTML output --------------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. -#html_theme = 'default' -#html_theme = "sphinxdoc" -html_theme = "sphinx_rtd_theme" -html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] - -def setup(app): - app.add_stylesheet("theme_overrides.css") # overrides for wide tables in RTD theme - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -#html_theme_options = {} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -#html_title = None - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -#html_logo = None -#html_logo = 'neuron.png' - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -#html_favicon = 'neuron-180x180.png' - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Output file base name for HTML help builder. -htmlhelp_basename = 'sampledoc' - - -# -- Options for LaTeX output -------------------------------------------------- - -latex_elements = { -# The paper size ('letterpaper' or 'a4paper'). -'papersize': 'letterpaper', - -# The font size ('10pt', '11pt' or '12pt'). -'pointsize': '10pt', - -# Additional stuff for the LaTeX preamble. -'preamble': '\setcounter{secnumdepth}{6}', -} - -# Grouping the document tree into LaTeX files. List of tuples -# (source start file, target name, title, author, documentclass [howto/manual]). -#latex_documents = [ -# ('index', 'sample.tex', u'sample Documentation', -# u'Kenneth Reitz', 'manual'), -#] - -# The name of an image file (relative to this directory) to place at the top of -# the title page. -#latex_logo = None - -# For "manual" documents, if this is true, then toplevel headings are parts, -# not chapters. -#latex_use_parts = False - -# If true, show page references after internal links. -#latex_show_pagerefs = False - -# If true, show URL addresses after external links. -#latex_show_urls = False - -# Documents to append as an appendix to all manuals. -#latex_appendices = [] - -# If false, no module index is generated. -#latex_domain_indices = True - - -# -- Options for manual page output -------------------------------------------- - -# One entry per manual page. List of tuples -# (source start file, name, description, authors, manual section). -#man_pages = [ -# ('index', 'sample', u'sample Documentation', -# [u'Kenneth Reitz'], 1) -#] -# -## If true, show URL addresses after external links. -##man_show_urls = False -# -# -## -- Options for Texinfo output ------------------------------------------------ -# -## Grouping the document tree into Texinfo files. List of tuples -## (source start file, target name, title, author, -## dir menu entry, description, category) -#texinfo_documents = [ -# ('index', 'sample', u'sample Documentation', -# u'Kenneth Reitz', 'sample', 'One line description of project.', -# 'Miscellaneous'), -#] - -# Documents to append as an appendix to all manuals. -#texinfo_appendices = [] - -# If false, no module index is generated. -#texinfo_domain_indices = True - -# How to display URL addresses: 'footnote', 'no', or 'inline'. -#texinfo_show_urls = 'footnote' diff --git a/docs/language/source/credits.rst b/docs/language/source/credits.rst deleted file mode 100644 index 32cfc29c..00000000 --- a/docs/language/source/credits.rst +++ /dev/null @@ -1,43 +0,0 @@ -******* -Credits -******* - -Acknowledgments -=============== - -For details on the partners, members, and supporters of NWB:N please the http://www.nwb.org/ project website. -For specific contributions to the format specification and this document see the change logs of -the Git repository at https://github.com/NeurodataWithoutBorders/nwb-schema . - -Authors -======= - -NWB:N: Version 2.0.0 and later ------------------------------- - -Documentation for Version 2 of the NWB:N specification and later have been created by -Oliver Ruebel and Andrew Tritt et al. in collaboration with the NWB:N community. - -NWB:N: Version 1.0.x and earlier --------------------------------- - -The specification language and corresponding documentation for Version 1.0.5g (and earlier) -of the NWB file format were created by Jeff Teeters et al. as -part of the first NWB pilot project. The documents for NWB:N 2 have been adopted from the -final version of format docs released by the original NWB pilot project. - - -***** -Legal -***** - -Copyright -========= - -.. include:: ../../../Legal.txt - -Licence -======= - -.. include:: ../../../license.txt - diff --git a/docs/language/source/index.rst b/docs/language/source/index.rst deleted file mode 100644 index f70f3641..00000000 --- a/docs/language/source/index.rst +++ /dev/null @@ -1,21 +0,0 @@ -Welcome to the NWB Specification Language -========================================= - - -.. toctree:: - :numbered: - :maxdepth: 6 - :caption: Table of Contents - - specification_language_description - specification_language_release_notes - credits - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - - diff --git a/docs/language/source/specification_language_description.rst b/docs/language/source/specification_language_description.rst deleted file mode 100644 index 93ad765d..00000000 --- a/docs/language/source/specification_language_description.rst +++ /dev/null @@ -1,966 +0,0 @@ -.. _specification_language: - -************************** -NWB Specification Language -************************** - -Version: |release| |today| [1]_ - -Introduction -============ - -In order to support the formal and verifiable specification of neurodata -file formats, NWB:N defines and uses the NWB specification language. -The specification language is defined in YAML (or optionally JSON) and defines formal -structures for describing the organization of complex data using basic -concepts, e.g., Groups, Datasets, Attributes, and Links. -A specification typically consists of a declaration of a namespace -and a set of schema specifications. -Data publishers can use the specification language to extend -the format in order to store types of data not supported by the -NWB core format (:numref:`sec-extensions`). - -.. seealso:: - - * The mapping of objects described in the specification language to HDF5 is - described in more detail in the NWB storage docs available here http://nwb-storage.readthedocs.io/en/latest/ - * Data structures for interacting with the specification language documents - (e.g, namespace and specification YAML/JSON files) are available as part of - PyNWB. For further details see the PyNWB docs available here: http://pynwb.readthedocs.io/en/latest/index.html - * For a general overview of the NWB:N data format see here: http://nwb-overview.readthedocs.io/en/latest/ - * For detailed description of the actual NWB:N data format see here: http://nwb-schema.readthedocs.io/en/latest/index.html - - -.. _sec-extensions: - -Extensions -========== - -As mentioned, extensions to the core format are specified via custom -user namespaces. Each namespace must have a unique name (i.e, must be -different from NWB). The schema of new neurodata_types (groups, datasets etc.) -are then specified in separate schema specification files. -While it is possible to define multiple namespaces in the same file, most commonly, -each new namespace will be defined in a separate file with corresponding -schema specifications being stored in one ore more additional YAML (or JSON) files. -One or more namespaces can be used simultaneously, so that multiple -extensions can be used at the same time while avoiding potential -name and type collisions between extensions (as well as extensions and the NWB core spec). - -The specification of namespaces is described in detail next in :numref:`sec-namespace-dec` -and the specification of schema specifications is described in :numref:`sec-schema-spec` -and subsequent sections. - -.. tip:: - - The ``form`` package as part of the PyNWB Python API provides dedicated - data structures and utilities that support programmatic generation of - extensions via Python programs, compared to writing YAML (or JSON) - extension documents by hand. One main advantage of using PyNWB is that it - is easier to use and maintain. E.g., using PyNWB helps ensure compliance of the - generated specification files with the current specification language and - the Python programs can often easily be just rerun to generate updated - versions of extension files (with little to no changes to the program itself). - -.. tip:: - - The ``hdmf-docutils`` package includes tools to generate Sphinx documentation from - format specifications. In particular the executable ``hdmf_init_sphinx_extension_doc`` - provides functionality to setup documentation for a format or extension defined - by a namespace (similar to the documentation for NWB core namespace at http://nwb-schema.readthedocs.io/en/latest/ ). - Use ``hdmf_init_sphinx_extension_doc --help`` to view the list - of options for generating the docs. The package also includes the executable ``hdmf_generate_format_docs`` - which is used for generating actual reStructuredText files and figures from YAML/JSON - specification sources. For an example see: http://pynwb.readthedocs.io/en/latest/example.html#documenting-extensions - -.. seealso:: - - For examples on how to create and use extensions in PyNWB see: - - * http://pynwb.readthedocs.io/en/latest/example.html#extending-nwb : Examples showing how to extend NWB - * http://pynwb.readthedocs.io/en/latest/tutorials.html#extensions : Tutorial showing how to define and use extensions - - - -.. _sec-namespace-dec: - -Namespaces -========== - -Namespaces are used to define a collections of specifications, to enable -users to develop extensions in their own namespace and, hence, to avoid -name/type collisions. Namespaces are defined in separate YAML files. -The specification of a namespace looks as follows: - -.. code-block:: python - - namespaces: - - doc: NWB namespace - name: NWB - full_name: NWB core - version: 1.2.0 - date: 2019-05-22 - author: - - Andrew Tritt - - Oliver Ruebel - - Ryan Ly - - Ben Dichter - - Keith Godfrey - - Jeff Teeters - contact: - - ajtritt@lbl.gov - - oruebel@lbl.gov - - rly@lbl.gov - - bdichter@lbl.gov - - keithg@alleninstitute.org - - jteeters@berkeley.edu - schema: - - source: nwb.base.yaml - neurodata_types: null - doc : Base nwb types - title : Base types - - ... - -The top-level key must be ``namespaces``. The value of ``namespaces`` -is a list with the specification of one (or more) namespaces. - - -Namespace declaration keys --------------------------- - -``doc`` -^^^^^^^ - -Text description of the namespace. - -``name`` -^^^^^^^^ - -Unique name used to refer to the namespace. - -``full_name`` -^^^^^^^^^^^^^ - -Optional string with extended full name for the namespace. - -``version`` -^^^^^^^^^^^ - -Version string for the namespace. - -``date`` -^^^^^^^^ - -Date the namespace has been last modified or released. Formatting is ``%Y-%m-%d %H:%M:%S``, e.g., ``2017-04-25 17:14:13``. - -``author`` -^^^^^^^^^^ - -List of strings with the names of the authors of the namespace. - -``contact`` -^^^^^^^^^^^ - -List of strings with the contact information for the authors. -Ordering of the contacts should match the ordering of the authors. - -``schema`` -^^^^^^^^^^ - -List of the schema to be included in this namespace. The specification looks as follows: - -.. code-block:: python - - - source: nwb.base.yaml - - source: nwb.ephys.yaml - doc: Types related to EPhys - title: EPhys - neurodata_types: - - ElectricalSeries - - namespace: core - neurodata_types: - - Interface - -* ``source`` describes the name of the YAML (or JSON) file with the schema specification. The schema files should be located in the same folder as the namespace file. -* ``namespace`` describes a named reference to another namespace. In contrast to source, this is a reference by name to a known namespace (i.e., the namespace is resolved during the build and must point to an already existing namespace). This mechanism is used to allow, e.g., extension of a core namespace (here the NWB core namespace) without requiring hard paths to the files describing the core namespace. -* ``neurodata_types`` then is an optional list of strings indicating which neurodata_types should be - included from the given specification source or namespace. The default is ``neurodata_types: null`` indicating that all - neurodata_types should be included. -* ``doc`` is an optional key for source files with a doc string to further document the content of the source file. -* ``title`` is an option key for source files to provide a descriptive title for a file for documentation purposes. - - -.. attention:: - - As with any language, we can only use what is defined. This means that similar to include or import statements in programming languages, e.g., Python, the ``source`` and ``namespace`` keys must be in order of use. E.g., ``nwb.ephys.yaml`` defines ``ElectricalSeries`` which inherits from ``Timeseries`` that is defined in ``nwb.base.yaml``. This means that we have to list ``nwb.base.yaml`` before ``nwb.ephys.yaml`` since otherwise ``Timeseries`` would not be defined when ``nwb.ephys.yaml`` is trying to use it. - - -.. _sec-schema-spec: - -Schema specification -==================== - -The schema specification defines the groups, datasets and -relationship that make up the format. Schema specifications are stored in dict ``spec`` and -consist of a list of Group specifications. -Schemas may be distributed across multiple YAML files to improve -readability and to support logical organization of types. -This is the main part of the format specification. It is described in the following sections. - -.. code-block:: yaml - - specs: - - ... - -.. note:: - - Schema specifications are agnostic to namespaces, i.e., a schema (or type) becomes - part of a namespace by including it in the namespace as part of the ``schema`` - description of the namespace. Hence, the same schema can be reused across - namespaces. - -.. _sec-group-spec: - -Groups -====== - -Groups are specified as part of the top-level list or via lists stored in the key -``groups``. The specification of a group is described in YAML as follows: - -.. code-block:: yaml - - - # Group specification - - neurodata_type_def: Optional new neurodata_type for the group - neurodata_type_inc: Optional neurodata_type the group should inherit from - name: Optional fixed name for the group. A group must either have a unique neurodata_type or a unique, fixed name. - default_name: Default name for the group - doc: Required description of the group - quantity: Optional quantity identifier for the group (default=1). - linkable: Boolean indicating whether the group is linkable (default=True) - attributes: Optional list of attribute specifications describing the attributes of the group - datasets: Optional list of dataset specifications describing the datasets contained in the group - groups: Optional list of group specifications describing the sub-groups contained in the group - links: Optional list of link specification describing the links contained in the group - -The key/value pairs that make up a group specification are described in more detail next in Section :numref:`sec-group-spec-keys`. -The keys should be ordered as specified above for readability and consistency with the rest of the schema. - -.. _sec-group-spec-keys: - -Group specification keys ------------------------- - -.. _sec-neurodata-type: - -``neurodata_type_def`` and ``neurodata_type_inc`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The concept of a neurodata_type is similar to the concept of Class in object-oriented programming. -A neurodata_type is a unique identifier for a specific type of group (or dataset) in a specification. -By assigning a neurodata_type to a group (or dataset) enables others to reuse that type by inclusion or -inheritance (*Note:* only groups (or datasets) with a specified type can be reused). - -- ```neurodata_type_def```: This key is used to define (i.e., create) a new neurodata_type and to assign that type to - the current group (or dataset). - -- ```neurodata_type_inc```: The value of the ``neurodata_type_inc`` key describes the base type - of a group (or dataset). The value must be an existing type. - -Both ```neurodata_type_def``` and ```neurodata_type_inc``` are optional keys. -To enable the unique identification, every group (and dataset) must either have a fixed name and/or a -unique neurodata_type. This means, any group (or dataset) with a variable name must have a unique neurodata_type. - -The neurodata_type is determined by the value of the ``neurodata_type_def`` key or if no new -type is defined then the value of ``neurodata_type_inc`` is used to determine type. Or in other -words, the neurodata_type is determined by the last type in the ancestry (i.e., inheritance hierarchy) of an object. - - -**Reusing existing neurodata_types** - -The combination of ```neurodata_type_inc``` and ```neurodata_type_def``` provides an easy-to-use mechanism for -reuse of type specifications via inheritance (i.e., merge and extension of specifications) and inclusion (i.e., -embedding of an existing type as a component, such as a subgroup, of a new specification). Here an overview -of all relevant cases: - -+------------------------+------------------------+------------------------------------------------------------------------+ -| ``neurodata_type_inc`` | ``neurodata_type_def`` | Description | -+========================+========================+========================================================================+ -|not set | not set | define a standard dataset or group without a type | -+------------------------+------------------------+------------------------------------------------------------------------+ -|not set | set | create a new neurodata_type from scratch | -+------------------------+------------------------+------------------------------------------------------------------------+ -|set | not set | include (reuse) neurodata_type without creating a new one (include) | -+------------------------+------------------------+------------------------------------------------------------------------+ -|set | set | merge/extend neurodata_type and create a new type (inheritance/merge) | -+------------------------+------------------------+------------------------------------------------------------------------+ - -**Example: Reuse by inheritance** - -.. code-block:: yaml - - # Abbreviated YAML specification - - neurodata_type_def: Series - datasets: - - name: A - - - neurodata_type_def: MySeries - neurodata_type_inc: Series - datasets: - - name: B - -The result of this is that ``MySeries`` inherits dataset ``A`` from ``Series`` and adds its own dataset ``B``, i.e., -if we resolve the inheritance, then the above is equivalent to: - -.. code-block:: yaml - - # Result: - - neurodata_type_def: MySeries - datasets: - - name: A - - name: B - -**Example: Reuse by inclusion** - -.. code-block:: yaml - - # Abbreviated YAML specification - - neurodata_type_def: Series - datasets: - - name: A - - - neurodata_type_def: MySeries - groups: - - neurodata_type_inc: Series - -The result of this is that ``MySeries`` now includes a group of type ``Series``, i.e., the above is equivalent to: - -.. code-block:: yaml - - - neurodata_type_def: MySeries - groups: - - neurodata_type_inc: Series - datasets: - - name: A - -.. note:: - - The keys ```neurodata_type_def`` and ```neurodata_type_inc``` were introduced in version 1.2a to - simplify the concepts of inclusion and merging of specifications and replaced the - keys ```include``` and ```merge```(and ```merge+```). - -``name`` -^^^^^^^^ - -String with the optional fixed name for the group. - -.. note:: - - Every group must have either a unique fixed ``name`` or a unique ``neurodata_type`` determined by - (``neurodata_type_def`` and ``neurodata_type_inc``) to enable the unique - identification of groups when stored on disk. - -``default_name`` -^^^^^^^^^^^^^^^^ - -Default name of the group. - -.. note:: - - Only one of either ``name`` or ``default_name`` (or neither) should be specified as the fixed - name given by ``name`` would always overwrite the behavior of ``default_name``. - -``doc`` -^^^^^^^ - -The value of the group specification ``doc`` key is a string -describing the group. The ``doc`` key is required. - -.. note:: - - In earlier versions (before version 1.2a) this key was called ``description`` - -.. _sec-quantity: - -``quantity`` -^^^^^^^^^^^^ - -The ``quantity`` describes how often the corresponding group (or dataset) can appear. The ``quantity`` -indicates both minimum and maximum number of instances. Hence, if the minimum number of instances is ``0`` -then the group (or dataset) is optional and otherwise it is required. The default value is ``quantity=1``. - -+---------------------------------+-------------------+------------------+--------------------------+ -| value | minimum quantity | maximum quantity | Comment | -+=================================+===================+==================+==========================+ -| ```zero_or_many``` or ```*``` | ``0`` | ``unlimited`` | Zero or more instances | -+---------------------------------+-------------------+------------------+--------------------------+ -| ```one_or_many``` or ```+``` | ``1`` | ``unlimited`` | One or more instances | -+---------------------------------+-------------------+------------------+--------------------------+ -| ```zero_or_one``` or ```?``` | ``0`` | ``1`` | Zero or one instances | -+---------------------------------+-------------------+------------------+--------------------------+ -| ```1```, ```2```, ```3```, ... | ``n`` | ``n`` | Exactly ``n`` instances | -+---------------------------------+-------------------+------------------+--------------------------+ - -.. note:: - - The ``quantity`` key was added in version 1.2a of the specification language as a replacement of the - ```quantity_flag``` that was used to encode quantity information via a regular expression as part of the - main key of the group. - -``linkable`` -^^^^^^^^^^^^ - -Boolean describing whether the this group can be linked. - -``attributes`` -^^^^^^^^^^^^^^ - -List of attribute specifications describing the attributes of the group. See :numref:`sec-attributes-spec` for details. - -.. code-block:: yaml - - attributes: - - ... - -``datasets`` -^^^^^^^^^^^^ - -List of dataset specifications describing all datasets to be stored as part of this group. -See :numref:`sec-dataset-spec` for details. - -.. code-block:: yaml - - datasets: - - name: data1 - doc: My data 1 - type: int - quantity: 'zero_or_one' - - name: data2 - doc: My data 2 - type: text - attributes: - - ... - - ... - -``groups`` -^^^^^^^^^^ - -List of group specifications describing all groups to be stored as part of this group - -.. code-block:: yaml - - groups: - - name: group1 - quantity: 'zero_or_one' - - ... - -``links`` -^^^^^^^^^ - -List of link specifications describing all links to be stored as part of this group. -See :numref:`sec-link-spec` for details. - -.. code-block:: yaml - - links: - - doc: Link to target type - name: link name - target_type: type of target - quantity: optional number of links allowed - - ... - -``\_required`` -^^^^^^^^^^^^^^ - -.. attention:: - - The ``\_required`` key has been removed in version 2.0. An improved version may - be added again in later version of the specification language. - - -.. _sec-attributes-spec: - -Attributes -========== - -Attributes are specified as part of lists stored in the key -``attributes`` as part of the specifications of ``groups`` and ``datasets``. -Attributes are typically used to further characterize or store metadata about -the group, dataset, or link they are associated with. Similar to datasets, attributes -can define arbitrary n-dimensional arrays, but are typically used to store smaller data. -The specification of an attributes is described in YAML as follows: - -.. code-block:: yaml - - ... - attributes: - - name: Required string describing the name of the attribute - dtype: Required string describing the data type of the attribute - dims: Optional list describing the names of the dimensions of the data array stored by the attribute (default=None) - shape: Optional list describing the allowed shape(s) of the data array stored by the attribute (default=None) - value: Optional constant, fixed value for the attribute. - default_value: Optional default value for variable-valued attributes. Only one of value or default_value should be set. - doc: Required string with the description of the attribute - required: Optional boolean indicating whether the attribute is required (default=True) - -The keys should be ordered as specified above for readability and consistency with the rest of the schema. - - -Attribute specification keys ----------------------------- - -``name`` -^^^^^^^^ - -String with the name for the attribute. The ``name`` key is required and must -specify a unique attribute on the current parent object (e.g., group or dataset) - -.. _sec-dtype: - -``dtype`` -^^^^^^^^^ - -String specifying the data type of the attribute. Allowable values are: - -+--------------------------+----------------------------------+----------------+ -| ``dtype`` **spec value** | **storage type** | **size** | -+--------------------------+----------------------------------+----------------+ -| * "float" | single precision floating point | 32 bit | -| * "float32" | | | -+--------------------------+----------------------------------+----------------+ -| * "double" | double precision floating point | 64 bit | -| * "float64" | | | -+--------------------------+----------------------------------+----------------+ -| * "long" | signed 64 bit integer | 64 bit | -| * "int64" | | | -+--------------------------+----------------------------------+----------------+ -| * "int" | signed 32 bit integer | 32 bit | -| * "int32" | | | -+--------------------------+----------------------------------+----------------+ -| * "short" | signed 16 bit integer | 16 bit | -| * "int16" | | | -+--------------------------+----------------------------------+----------------+ -| * "int8" | signed 8 bit integer | 8 bit | -+--------------------------+----------------------------------+----------------+ -| * "uint64" | unsigned 64 bit integer | 64 bit | -+--------------------------+----------------------------------+----------------+ -| * "uint32" | unsigned 32 bit integer | 32 bit | -+--------------------------+----------------------------------+----------------+ -| * "uint16" | unsigned 16 bit integer | 16 bit | -+--------------------------+----------------------------------+----------------+ -| * "uint8" | unsigned 8 bit integer | 8 bit | -+--------------------------+----------------------------------+----------------+ -| * "numeric" | any numeric type (i.e., int, | 8 to 64 bit | -| | uint, float etc.) | | -+--------------------------+----------------------------------+----------------+ -| * "text" | 8-bit Unicode | variable | -| * "utf" | | | -| * "utf8" | | | -| * "utf-8" | | | -+--------------------------+----------------------------------+----------------+ -| * "ascii" | ASCII text | variable | -| * "bytes" | | | -+--------------------------+----------------------------------+----------------+ -| * "bool" | 8 bit integer with valid values | 8 bit | -| | 0 or 1 | | -+--------------------------+----------------------------------+----------------+ -| * "isodatetime" | ISO8061 datetime string, e.g., | variable | -| | 2018-09-28T14:43:54.123+02:00 | | -| * "datetime" | | | -+--------------------------+----------------------------------+----------------+ - -.. note:: - - The precision indicated in the specification is generally interpreted as a minimum precision. - Higher precisions may be used if required by the particular data. - In addition, since valid ASCII text is valid UTF-8-encoded Unicode, ASCII text may be used - where 8-bit Unicode is required. 8-bit Unicode cannot be used where ASCII is required. - - -Reference ``dtype`` -""""""""""""""""""" - -In addition to the above basic data types, an attribute or dataset may also store references to other -data objects. Reference ``dtypes`` are described via a dictionary. E.g.: - -.. code-block:: yaml - - dtype: - target_type: ElectrodeGroup - reftype: object - - -``target_type`` here describes the ``neurodata_type`` of the target that the reference points to and -``reftype`` describes the kind of reference. Currently the specification language supports two main -reference types. - - -+--------------------------+-------------------------------------+ -| ``reftype`` **value** | **Reference type description** | -+--------------------------+-------------------------------------+ -| * "ref" | Reference to another group or | -| * "reference" | dataset of the given ` | -| * "object" | ``target_type`` | -+--------------------------+-------------------------------------+ -| * region | Reference to a region (i.e. subset) | -| | of another dataset of the given | -| | ``target_type`` | -+--------------------------+-------------------------------------+ - -Compound ``dtype`` -"""""""""""""""""" - -Compound data types are essentially a ``struct``, i.e., the data type is a composition of several primitive types. -This is useful to specify complex types, e.g., for storage of complex numbers consisting of a real and imaginary components, -vectors or tensors, as well to create table-like data structures. Compound data types are created by defining a list of -the form: - -.. code-block:: yaml - - dtype: - - name: - dtype: - doc: - - name: .... - . - . - . - -.. note:: - - Currently only "flat" compound types are allowed, i.e., a compound type may not contain other compound types - but may itself only consist of basic dtypes, e.g,. float, string, etc. or reference dtypes. - - -Below and example form the NWB:N format specification showing the use of compound data types to create a table-like -data structure for storing metadata about electrodes. - -.. code-block:: yaml - - datasets: - - doc: 'a table for storing queryable information about electrodes in a single table' - dtype: - - name: id - dtype: int - doc: a user-specified unique identifier - - name: x - dtype: float - doc: the x coordinate of the channels location - - name: y - dtype: float - doc: the y coordinate of the channels location - - name: z - dtype: float - doc: the z coordinate of the channels location - - name: imp - dtype: float - doc: the impedance of the channel - - name: location - dtype: ascii - doc: the location of channel within the subject e.g. brain region - - name: filtering - dtype: ascii - doc: description of hardware filtering - - name: description - dtype: utf8 - doc: a brief description of what this electrode is - - name: group - dtype: ascii - doc: the name of the ElectrodeGroup this electrode is a part of - - name: group_ref - dtype: - target_type: ElectrodeGroup - reftype: object - doc: a reference to the ElectrodeGroup this electrode is a part of - attributes: - - doc: Value is 'a table for storing data about extracellular electrodes' - dtype: text - name: help - value: a table for storing data about extracellular electrodes - neurodata_type_inc: NWBData - neurodata_type_def: ElectrodeTable - - -.. _sec-dims: - -``dims`` -^^^^^^^^ - -Optional key describing the names of the dimensions of the array stored as value of the attribute. -If the attribute stores an array, ``dims`` specifies the -list of dimensions. If no ``dims`` is given, then attribute stores a scalar value. - -In case there is only one option for naming the dimensions, the key defines -a single list of strings: - -.. code-block:: yaml - - ... - dims: - - dim1 - - dim2 - -In case that the attribute may have different forms, this will be a list of lists: - -.. code-block:: yaml - - ... - dims: - - - num_times - - - num_times - - num_channels - -Each entry in the list defines an identifier/name of the corresponding dimension -of the array data. - -.. _sec-shape: - -``shape`` -^^^^^^^^^ - -Optional key describing the shape of the array stored as the value of the attribute. -The description of ``shape`` must match the description of dimensions in so far as -if we name two dimensions in ``dims`` than we must also specify the ``shape`` for -two dimensions. We may specify ``null`` in case that the length of a dimension is not -restricted, e.g.: - -.. code-block:: yaml - - ... - shape: - - null - - 3 - -Similar to ``dims`` shape may also be a list of lists in case that the attribute -may have multiple valid shape options, e.g,: - -.. code-block:: yaml - - ... - shape: - - - 5 - - - null - - 5 - -The default behavior for shape is: - -.. code-block:: yaml - - ... - shape: null - -indicating that the attribute/dataset is a scalar. - -``value`` -^^^^^^^^^ - -Optional key specifying a fixed, constant value for the attribute. Default value is None, i.e., -the attribute has a variable value to be determined by the user (or API) in accordance with -the current data. - -.. _sec-default_value: - -``default_value`` -^^^^^^^^^^^^^^^^^ - -Optional key specifying a default value for attributes that allow user-defined values. The -default value is used in case that the user does not specify a specific value for the attribute. - -.. note:: - Only one of either ``value`` or ``default_value`` should be specified (or neither) but never - both at the same time, as ``value`` would always overwrite the ``default_value``. - -``doc`` -^^^^^^^ - -``doc`` specifies the documentation string for the attribute and should describe the -purpose and use of the attribute data. The ``doc`` key is required. - -``required`` -^^^^^^^^^^^^ - -Optional boolean key describing whether the attribute is required. Default value is True. - -.. _sec-value: - - -.. _sec-link-spec: - -Links -===== - -The link specification is used to specify links to other groups or datasets. -The link specification is a dictionary with the following form: - -.. code-block:: yaml - - links: - - name: link name - target_type: type of target - doc: Link to target type - -.. note:: - - When mapped to storage, links should always remain identifiable as such. For example, - in the context of HDF5, this means that soft links (or external links) should be - used instead of hard links. - -The keys should be ordered as specified above for readability and consistency with the rest of the schema. - -Link specification keys ------------------------- - -``name`` -^^^^^^^^ - -Optional key specifying the ``name`` of the link. - -``target_type`` -^^^^^^^^^^^^^^^ - -``target_type`` specifies the key for a group in the top level structure -of a namespace. It is used to indicate that the link must be to an -instance of that structure. - -``doc`` -^^^^^^^ - -``doc`` specifies the documentation string for the link and should describe the -purpose and use of the linked data. The ``doc`` key is required. - -``quantity`` -^^^^^^^^^^^^ - -Optional key specifying how many allowable instances for that link. Default is 1. If `name` is defined, quantity may -not be >1. See :numref:`sec-quantity` for details. - - -.. _sec-dataset-spec: - -Datasets -======== - -Datasets are specified as part of lists stored in the key ``datasets`` as part of group specifications. -The specification of a datasets is described in YAML as follows: - -.. code-block:: yaml - - - datasets: - - neurodata_type_def: Optional new neurodata_type for the group - neurodata_type_inc: Optional neurodata_type the group should inherit from - name: fixed name of the dataset - default_name: default name of the dataset - dtype: Optional string describing the data type of the dataset - dims: Optional list describing the names of the dimensions of the dataset - shape: Optional list describing the shape (or possible shapes) of the dataset - value: Optional to fix value of dataset - default_value: Optional to set a default value for the dataset - doc: Required description of the dataset - quantity: Optional quantity identifier for the group (default=1). - linkable: Boolean indicating whether the group is linkable (default=True) - attributes: Optional list of attribute specifications describing the attributes of the group - -The specification of datasets looks quite similar to attributes and groups. Similar to -attributes, datasets describe the storage of arbitrary n-dimensional array data. -However, in contrast to attributes, datasets are not associated with a specific parent -group or dataset object but are (similar to groups) primary data objects (and as such -typically manage larger data than attributes). -The key/value pairs that make up a dataset specification are described in more detail next in Section -:numref:`sec-dataset-spec-keys`. The keys should be ordered as specified above for readability and consistency with the -rest of the schema. - -.. _sec-dataset-spec-keys: - -Dataset specification keys --------------------------- - -``neurodata_type_inc`` and ``neurodata_type_def`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Same as for groups. See :numref:`sec-neurodata-type` for details. - -``name`` -^^^^^^^^ - -String with the optional fixed name for the dataset - -.. note:: - - Every dataset must have either a unique fixed ``name`` or a unique ``neurodata_type`` to enable the unique - identification of datasets when stored on disk. - -``default_name`` -^^^^^^^^^^^^^^^^ - -Default name of the group. - -.. note:: - - Only one of either ``name`` or ``default_name`` (or neither) should be specified as the fixed - name given by ``name`` would always overwrite the behavior of ``default_name``. - -``dtype`` -^^^^^^^^^ - -String describing the data type of the dataset. Same as for attributes. See :numref:`sec-dtype` for details. ``dtype`` may be omitted for abstract classes. Best practice is to define ``dtype`` for most concrete classes. - -``dims`` -^^^^^^^^ - -List describing the names of the dimensions of the dataset. Same as for attributes. See :numref:`sec-dims` for details. - -``shape`` -^^^^^^^^^ - -List describing the shape of the dataset. Same as for attributes. See :numref:`sec-shape` for details. - -``value`` and ``default_value`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Same as for attributes. See :numref:`sec-value` and :numref:`sec-default_value` for details. - -``doc`` -^^^^^^^ - -The value of the dataset specification ``doc`` key is a string -describing the dataset. The ``doc`` key is required. - -.. note:: - - In earlier versions (before version 1.2a) this key was called ``description`` - -``quantity`` -^^^^^^^^^^^^ - -Same as for groups. See :numref:`sec-quantity` for details. - -``linkable`` -^^^^^^^^^^^^ - -Boolean describing whether the this group can be linked. - -``attributes`` -^^^^^^^^^^^^^^ - -List of attribute specifications describing the attributes of the group. See Section :ref:`sec-attributes-spec` for details. - -.. code-block:: yaml - - attributes: - - ... - -Relationships -============= - -.. note:: - - Future versions will add explicit concepts for modeling of relationships, to replace the - implicit relationships encoded via shared dimension descriptions and implicit references in - datasets in previous versions of the specification language. - - - -.. [1] - The version number given here is for the specification language and - is independent of the version number for the NWB format. The date - after the version number is the last modification date of this - document. diff --git a/docs/language/source/specification_language_release_notes.rst b/docs/language/source/specification_language_release_notes.rst deleted file mode 100644 index 668b756a..00000000 --- a/docs/language/source/specification_language_release_notes.rst +++ /dev/null @@ -1,189 +0,0 @@ -============= -Release Notes -============= - -Version 2.0.2 (Upcoming) -------------------------- -* add ``value`` and ``default_value`` as optional keys of a dataset. -* ``dtype`` changed from required to optional for datasets. -* add dtypes that are already supported in `hdmf.spec`: short (int16), uint64, bytes (ascii), and datetime (isodatetime) - -Version 2.0.1 (March, 2019) ---------------------------------- -* Added support for specifying a ``title`` and ``doc`` for ``source`` files as part of the ``schema`` portion of a ``namespace`` specification. This was added to improve documentation of individual source files and to support sorting of types by source file with meaningful titles and text as part of autogenerated docs. -* Updated the docs for ``quantity`` to indicate that the default value is ``1`` if not specified. - - -Version 2.0.0 (January, 2019) ----------------------------------- - -Summary -^^^^^^^ -* Simplify reuse of neurodata_types: - * Added new key: ```neurodata_type_def and ```neurodata_type_inc``` (which in combination replace the keys ```neurodata_type```, ```include``` and ```merge```). See below for details. - * Removed key: ```include``` - * Removed key: ```merge``` - * Removed key: ```merge+``` - * Removed key: ```neurodata_type``` (replaced by ``neurodata_type_inc`` and ``neurodata_type_def``) - * Removed ```\_properties``` key. The primary use of the key is to define ``abstract`` specifications. However, as format specifications don't implement functions but define a layout of objects, any spec (even if marked abstract) could still be instantiated and used in practice without limitations. Also, in the current instantiation of NWB:N this concept is only used for the ```Interface``` type and it is unclear why a user should not be able to use it. As such this concept was removed. - * To improve compliance of NWB:N inheritance mechanism with established object-oriented design concepts, the option of restricting the use of subclasses in place of parent classes was removed. A subclass is always also a valid instance of a parent class. This also improves consistency with the NWB:N principle of a minimal specification that allows users to add custom data. This change affects the ```allow_subclasses``` key of links and the subclasses option of the removed ```include`` key. -* Improve readability and avoid collision of keys by replacing values encoded in keys with dedicated key/value pairs: - * Explicit encoding of names and types: - * Added ```name``` key - * Removed `<...>` name identifier (replaced by empty ```name``` key) - * Added ```groups``` key (previously groups were indicated by "/" as part of object's key) - * Added ```datasets``` key (previously datasets were indicated by missing "/" as part of the object's key) - * Added ```links``` key (previously this was a key on the group and dataset specification). The concept of links is with this now a first-class type (rather than being part of the group and dataset specs). - * Removed ``link`` key on datasets as this functionality is now fully implemented by the ``links`` key on groups. - * Removed `/` flag in keys to identify groups (replaced by ```groups``` and ```datasets``` keys) - * Explicit encoding of quantitites: - * Added new key ```quantity``` (which replaces the ```quantity_flag```). See below for details. - * Removed ```quantity_flag``` as part of keys - * Removed `Exclude\_in`` key. The key is currently not used in the NWB core spec. This feature is superseded by the ability to overwrite the ```quantity``` key as part of the reuse of ```neurodata_types``` - * Removed ```\_description``` key. The key is no longer need because name conflicts with datasets and groups are no longer possible since the name is now explicitly encoded in a dedicated key/value pair. -* Improve human readability: - * Added support for YAML in addition to JSON - * Values, such as, names, types, quantities etc. are now explicitly encoded in dedicated key/value pairs rather than being encoded as regular expressions in keys. -* Improve direct interpretation of data: - * Remove ```references``` key. This key was used in previous versions of NWB to generate implicit data structures where datasets store references to part of other metadata structures. These implicit data structures violate core NWB principles as they hinder the direct interpretation of data and cannot be interpreted (neither by human nor program) based on NWB files alone without having additional information about the specification as well. Through simple reorganization of metadata in the file, all instances of these implicit data structures were replaced by simple links that can be interpreted directly. -* Simplified specification of dimensions for datasets: - * Renamed ```dimensions``` key to ```dims``` - * Added key ```shape``` to allow the specification of the shape of datasets - * Removed custom keys for defining structures as types for dimensions: - * ```unit``` keys from previous structured dimensions are now ```unit``` attributes on the datasets (i.e., all values in a dataset have the same units) - * The length of the structs are used to define the length of the corresponding dimension as part of the ```shape``` key - * ```alias``` for components of dimensions are currently encoded in the dimensions name. -* Added support for default vs. fixed name for groups and datasets: - * Added ``default_name`` key for groups and dataset to allow the specification of default names for objects that can have user-defined names (in addition to fixed names via ``name``). Attributes can only have a fixed name since attributes can not have a neurodata_type and can, hence, only be identified via their fixed name. -* Updated specification of fixed and default values for attributes to make the behavior of keys explicit: - * Specifying attribute values: - * Added ``default_value`` key for attributes to specify a default value for attributes - * Removed ``const`` key for attributes which was used to control the behavior of the ``value`` key, i.e., depending on the value of ``const`` the ``value`` key would either act as a fixed or default value. By adding the ``default_value`` key this behavior now becomes explicit and the behavior of the ``value`` key no longer depends on the value of another key (i.e., the ``const`` key) -* Improved governance and reuse of specifications: - * The core specification documents are no longer stored as .py files as part of the original Python API but are released as separate YAML (or optionally JSON) documents in a seperate repository - * All documentation has been ported to use reStructuredText (RST) markup that can be easily translated to PDF, HTML, text, and many other forms. - * Documentation for source codes and the specification are auto-generated from source to ensure consistency between sources and the documentation -* Avoid mixing of format specification and computations: - * Removed key ```autogen``` (without replacement). The autogen key was used to describe how to compute certain derived datasets from the file. This feature was problematic with respect to the guiding principles of NWB for a couple of reasons. E.g., the resulting datasets were often not interpretable without the provenance of the autogeneration procedure and autogeneration itself and often described the generation of derived data structures to ease follow-on computations. Describing computations as part of a format specification is problematic as it creates strong dependencies and often unnecessary restrictions for use and analysis of data stored in the format. Also, the reorganization of metadata has eliminated the need for autogen in many cases. A autogen features is arguably the role of a data API or intermediary derived-quantity API (or specification), rather than a format specification. -* Enhanced specification of data types via ``dtype``: - * Enhanced the syntax for ``dtype`` to allow the specification of flat compound data types via lists of types - * Enhanced the syntax for ``dtype`` to allow the specification of i) object references and ii) region references - * Removed "!" syntax (e.g., "float32!") previously used to specify a minimum precision. All types are interpreted as minimum specs. - * Specified list of available data types and their names - * Added ``isodatetime`` dtype for specification of ISO8061 datetime string (e.g., ``2018-09-28T14:43:54.123+02:00``) as data type - * Added ``bool`` dtype for specification fo boolean type fields (see `PR691 (PyNWB) `_ - and `I658 (PyNWB) `_. -* Others: - * Removed key ```\_\_custom``` (without replacement). This feature was used only in one location to provide user hints where custom data could be placed, however, since the NWB specification approach explicitly allows users to add custom data in any location, this information was not binding. - - -Currently unsupported features: -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -* ```_required``` : The current API does not yet support specification and verification of constraints previously expressed via ``_required``. -* Relationships are currently available only through implicit concepts, i.e., by sharing dimension names and through implicit references as part of datasets. The goal is to provide explicit mechanisms for describing these as well as more advanced relationships. -* ```dimensions_specification```: This will be implemented in later version likely through the use of relationships. - - -YAML support -^^^^^^^^^^^^ - -To improve human readability of the specification language, Version 1.2a now allows specifications to be defined in YAML as well as JSON (Version 1.1c allowed only JSON). - -```quantity``` -^^^^^^^^^^^^^^ - - -Version 1.1c of the specification language used a ```quantity_flag``` as part of the name key of groups and datasets to the quantity - -* `!` - Required (this is the default) -* `?`- Optional -* `^` - Recommended -* `+` - One or more instances of variable-named identifier required -* `*` - Zero or more instances of variable-named identifier allowed - -Version 1.2a replaces the ```quantity_flag``` with a new key ```quantity``` with the following values: - -+---------------------------------+------------+-------------------------------------------------------+ -| value | required | number of instances | -+=================================+============+=======================================================+ -| ```zero_or_more``` or ```*``` | optional | unlimited | -+---------------------------------+------------+-------------------------------------------------------+ -| ```one_or_more``` or ```+``` | required | unlimited but at least 1 | -+---------------------------------+------------+-------------------------------------------------------+ -| ```zero_or_one``` or ```?``` | optional | 0 or 1 | -+---------------------------------+------------+-------------------------------------------------------+ -| ```1```, ```2```, ```3```, ... | required | Fixed number of instances as indicated by the value | -+---------------------------------+------------+-------------------------------------------------------+ - - -```merge``` and ```include``` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -To simplify the concept ```include``` and ```merge```, version 1.2a introduced a new -key ```neurodata_type_def``` which describes the creation of a new neurodata_type. -The combination ```neurodata_type_def``` and ```neurodata_type_inc`` -simplifies the concepts of merge (i.e., inheritance/extension) and inclusion and -allows us to express the same concepts in an easier-to-use fashion. -Accordingly, the keys ```include```, ```merge``` and ```merge+``` have been removed in version 1.2a. -Here a summary of the basic cases: - -+--------------------+--------------------+------------------------------------------------------------------------+ -| neurodata_type_inc | neurodata_type_def | Description | -+====================+====================+========================================================================+ -|not set | not set | define standard dataset or group without a type | -+--------------------+--------------------+------------------------------------------------------------------------+ -|not set | set | create a new neurodata_type from scratch | -+--------------------+--------------------+------------------------------------------------------------------------+ -|set | not set | include (reuse) neurodata_type without creating a new one (include) | -+--------------------+--------------------+------------------------------------------------------------------------+ -|set | set | merge/extend neurodata_type and create a new type (merge) | -+--------------------+--------------------+------------------------------------------------------------------------+ - -```structured_dimensions``` -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The definition of structured dimensions has been removed in version 1.2a. The concept of structs as dimensions is -problematic for several reasons: 1) it implies support for defining general tables with mixed units and data types -which are currently not supported, 2) they easily allow for colliding specification where mixed units are assigned -to the same value, 3) they are hard to use and unsupported by HDF5. Currently structured dimensions, however, have -been used only to encode information about "columns" of a dataset (e.g., to indicate that a dimension stores x,y,z -values). This information was translated to the ``dims``` and ```shape``` keys and ```unit``` attributes. -The more general concept of structured dimensions will be implemented in future versions of the specification language -and format likely via support for modeling of relationships or support for table data structures (stay tuned) - -```autogen``` -^^^^^^^^^^^^^ - -The ```autogen``` key has been removed without replacement. - -**Reason:** The autogen specification was originally used to specify that the attribute or dataset contents (values) -can be derived from the contents of the HDF5 file and, hence, generated and validated automatically. -As such, autogen crossed a broad range of different functionalities, including: - -1. Specification of the structure of format datasets/attributes -2. Description of data constraints (e.g., the shape of the generated dataset directly depends on the structure of the input data consumed by autogen), -3. Specification of the content (i.e., value) of datasets and attributes, -4. Description of computations to create derived data, and -5. Validation of the structure and content of datasets/attributes. - -This mixing of functionality in turn led to several concerns: - -* autogen exhibited a fairly complex syntax, which made it hard to interpret and use -* autogen is specifically used to create derived data from information that is already in the NWB file. - Attributes/datasets generated via autogen: i) are redundant, ii) often require bookkeeping to ensure data consistency, - iii) generate dependencies across data and types, iv) have limited utility as the information can be derived through - other means, and v) interpretation of data values may require the provenance of autogen. -* Description of computations as part of a format specification was seen as problematic. -* There was potential for collisions between autogen and the specification of the dataset/attribute itself. - -**Usage in NWB** autogen was used in NWB V.1.0.6 to generate 17 datasets/attributes primarily to: -i) store the path of links in separate datasets/attributes or ii) generate lists of datasets/groups of a given type/property. -The datasets were reviewed at a hackathon and determined to be non-essential and as such removed from the format as well. - - - -Version 1.1c (Oct. 7, 2016) ---------------------------- - -* Original version of the specification language generated as part of the NWB pilot project diff --git a/license.txt b/license.txt index 38f5ec81..fe86e283 100644 --- a/license.txt +++ b/license.txt @@ -1,4 +1,4 @@ -“nwb-schema” Copyright (c) 2017-2020, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. +“nwb-schema” Copyright (c) 2017-2021, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: