From 22bbd253f4e07a810a3e4f4dde63ad66508cc9d3 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 11 Sep 2024 09:55:12 +0200 Subject: [PATCH 1/9] Unify rfc 1 and 2 layouts and tables --- rfc/1/index.md | 44 ++++-- rfc/2/index.md | 143 ++++++++++++++++---- rfc/2/{review_1.md => reviews/1/index.md} | 0 rfc/2/{review_1b.md => reviews/1b/index.md} | 0 rfc/2/{review_2.md => reviews/2/index.md} | 0 rfc/2/{review_3.md => reviews/3/index.md} | 0 6 files changed, 148 insertions(+), 39 deletions(-) rename rfc/2/{review_1.md => reviews/1/index.md} (100%) rename rfc/2/{review_1b.md => reviews/1b/index.md} (100%) rename rfc/2/{review_2.md => reviews/2/index.md} (100%) rename rfc/2/{review_3.md => reviews/3/index.md} (100%) diff --git a/rfc/1/index.md b/rfc/1/index.md index f1ba15d8..dee12599 100644 --- a/rfc/1/index.md +++ b/rfc/1/index.md @@ -18,76 +18,90 @@ versions/index This RFC is currently being responded to (R4). ```{list-table} Record -:widths: 20, 20, 20, 15, 10 +:widths: 8, 20, 20, 20, 15, 10 :header-rows: 1 :stub-columns: 1 -* - Name +* - Role + - Name - GitHub Handle - Institution - Date - Status -* - Josh Moore +* - Author + - Josh Moore - [joshmoore](https://github.com/joshmoore) - [German BioImaging, e.V.](https://ror.org/05tpnw772) - 2023-12-23 - Author ([PR](https://github.com/ome/ngff/pull/222)) -* - Davis Bennett, John Bogovic, Michael Innerberger, Mark Kittisopikul, Virginia Scarlett, Yurii Zubov +* - Reviwer + - Davis Bennett, John Bogovic, Michael Innerberger, Mark Kittisopikul, Virginia Scarlett, Yurii Zubov - d-v-b, bogovicj, minnerbe, mkitti, virginiascarlett, yuriyzubov - Janelia - 2024-02-26 - [Review](./reviews/2/index) -* - Kevin Yamauchi, Virginie Uhlmann +* - Reviwer + - Kevin Yamauchi, Virginie Uhlmann - kevinyamauchi, vuhlmann - ETH, BiovisionCenter - 2024-03-05 - [Review](./reviews/1/index) -* - Matthew Hartley +* - Reviwer + - Matthew Hartley - mrmh2 - EMBL-EBI - 2024-03-05 - [Review](./reviews/3/index) -* - Wouter-Michiel Vierdag, Luca Marconato +* - Commenter + - Wouter-Michiel Vierdag, Luca Marconato - melonora, LucaMarconato - EMBL - 2024-01-13 - [Comment](./comments/1/index) -* - Matt McCormick +* - Commenter + - Matt McCormick - thewtex - ITK - 2024-01-09 - [Comment](./comments/2/index) -* - Josh Moore +* - Author + - Josh Moore - joshmoore - German BioImaging - 2024-08-29 - [Response](./responses/1/index) -* - Jean-Marie Burel +* - Endorser + - Jean-Marie Burel - jburel - University of Dundee - 2024-09-09 - [Endorse](https://github.com/ome/ngff/pull/258) -* - Will Moore +* - Endorser + - Will Moore - will-moore - University of Dundee - 2024-09-09 - [Endorse](https://github.com/ome/ngff/pull/258) -* - Juan-Nuñez Iglesias +* - Endorser + - Juan-Nuñez Iglesias - jni - Monash University - 2024-09-09 - [Endorse](https://github.com/ome/ngff/pull/258) -* - Norman Rzepka +* - Endorser + - Norman Rzepka - normanrz - scalable minds GmbH - 2024-09-09 - [Endorse](https://github.com/ome/ngff/pull/258) -* - Jan Eglinger +* - Endorser + - Jan Eglinger - imagejan - FMI Basel - 2024-09-09 - [Endorse](https://github.com/ome/ngff/pull/258) -* - Joel Lüthi +* - Endorser + - Joel Lüthi - jluethi - BioVisionCenter, University of Zurich - 2024-09-10 diff --git a/rfc/2/index.md b/rfc/2/index.md index dda8c466..49b14f32 100644 --- a/rfc/2/index.md +++ b/rfc/2/index.md @@ -1,12 +1,12 @@ # RFC-2: Zarr v3 ```{toctree} -:maxdepth: 1 :hidden: -review_1 -review_1b -review_2 -review_3 +:maxdepth: 1 +reviews/index +comments/index +responses/index +versions/index ``` Adopt the version 3 of Zarr for OME-Zarr. @@ -15,25 +15,120 @@ Adopt the version 3 of Zarr for OME-Zarr. This RFC is currently in RFC state (R6). -| Role | Name | GitHub Handle | Institution | Date | Status | -| -------- | ------------------- | --------------------------------------------------- | -------------------------------------------------- | ---------- | ----------------------------------------------------------------------- | -| Author | Norman Rzepka | [normanrz](https://github.com/normanrz) | [scalable minds](https://scalableminds.com) | 2024-02-14 | | -| Endorser | Davis Bennett | [d-v-b](https://github.com/d-v-b) | | 2024-02-14 | Endorse | -| Endorser | Kevin Yamauchi | [kevinyamauchi](https://github.com/kevinyamauchi) | ETH Zürich | 2024-02-16 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1947942934) | -| Endorser | John Bogivic | [bogovicj](https://github.com/bogovicj) | HHMI Janelia Research Campus | 2024-02-16 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1948547356) | -| Endorser | Matthew Hartley | [matthewh-ebi](https://github.com/matthewh-ebi) | EMBL-EBI | 2024-02-16 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1948912814) | -| Endorser | Christian Tischer | [tischi](https://github.com/tischi) | EMBL | 2024-02-16 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1949058616) | -| Endorser | Joel Lüthi | [jluethi](https://github.com/jluethi) | BioVisionCenter, University of Zurich | 2024-02-16 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1949333769) | -| Endorser | Constantin Pape | [constantinpape](https://github.com/constantinpape) | University Göttingen | 2024-02-18 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1951318754) | -| Endorser | Will Moore | [will-moore](https://github.com/will-moore) | OME, University of Dundee | 2024-02-19 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1952057704) | -| Endorser | Juan Nunez-Iglesias | [jni](https://github.com/jni) | Biomedicine Discovery Institute, Monash University | 2024-02-20 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1953922897) | -| Endorser | Eric Perlman | [perlman](https://github.com/perlman) | | 2024-02-22 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1960272942) | -| Endorser | Ziwen Liu | [ziw-liu](https://github.com/ziw-liu) | Chan Zuckerberg Biohub | 2024-03-12 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1992588774) | -| Endorser | Lachlan Deakin | [LDeakin](https://github.com/LDeakin) | Australian National University | 2024-03-14 | [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1998594492) | -| Reviewer | Melissa Linkert, Sebastién Besson, Chris Allan, Jason Swedlow | glencoesoftware | Glencoe Software | 2024-05-23 | [Review](./review_1.md) | -| Reviewer | Yaroslav O. Halchenko | yarikoptic | Dartmouth College, DANDI Project | 2024-06-10 | [Review](./review_2.md) | -| Reviewer | Jeremy Maitin-Shepard | jbms | Google | 2024-04-30 | [Review](./review_3.md) | -| Reviewer | Melissa Linkert, Sebastién Besson, Chris Allan, Jason Swedlow | glencoesoftware | Glencoe Software | 2024-08-05 | [Accept](./review_1b.md) | +```{list-table} Record +:widths: 8, 20, 20, 20, 15, 10 +:header-rows: 1 +:stub-columns: 1 + +* - Role + - Name + - GitHub Handle + - Institution + - Date + - Status +* - Author + - Norman Rzepka + - [normanrz](https://github.com/normanrz) + - [scalable minds](https://scalableminds.com) + - 2024-02-14 + - +* - Endorser + - Davis Bennett + - [d-v-b](https://github.com/d-v-b) + - + - 2024-02-14 + - Endorse +* - Endorser + - Kevin Yamauchi + - [kevinyamauchi](https://github.com/kevinyamauchi) + - ETH Zürich + - 2024-02-16 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1947942934) +* - Endorser + - John Bogivic + - [bogovicj](https://github.com/bogovicj) + - HHMI Janelia Research Campus + - 2024-02-16 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1948547356) +* - Endorser + - Matthew Hartley + - [matthewh-ebi](https://github.com/matthewh-ebi) + - EMBL-EBI + - 2024-02-16 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1948912814) +* - Endorser + - Christian Tischer + - [tischi](https://github.com/tischi) + - EMBL + - 2024-02-16 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1949058616) +* - Endorser + - Joel Lüthi + - [jluethi](https://github.com/jluethi) + - BioVisionCenter, University of Zurich + - 2024-02-16 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1949333769) +* - Endorser + - Constantin Pape + - [constantinpape](https://github.com/constantinpape) + - University Göttingen + - 2024-02-18 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1951318754) +* - Endorser + - Will Moore + - [will-moore](https://github.com/will-moore) + - OME, University of Dundee + - 2024-02-19 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1952057704) +* - Endorser + - Juan Nunez-Iglesias + - [jni](https://github.com/jni) + - Biomedicine Discovery Institute, Monash University + - 2024-02-20 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1953922897) +* - Endorser + - Eric Perlman + - [perlman](https://github.com/perlman) + - + - 2024-02-22 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1960272942) +* - Endorser + - Ziwen Liu + - [ziw-liu](https://github.com/ziw-liu) + - Chan Zuckerberg Biohub + - 2024-03-12 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1992588774) +* - Endorser + - Lachlan Deakin + - [LDeakin](https://github.com/LDeakin) + - Australian National University + - 2024-03-14 + - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1998594492) +* - Reviewer + - Melissa Linkert, Sebastién Besson, Chris Allan, Jason Swedlow + - glencoesoftware + - Glencoe Software + - 2024-05-23 + - [Review](./reviews/1/index) +* - Reviewer + - Yaroslav O. Halchenko + - yarikoptic + - Dartmouth College, DANDI Project + - 2024-06-10 + - [Review](./reviews/2/index) +* - Reviewer + - Jeremy Maitin-Shepard + - jbms + - Google + - 2024-04-30 + - [Review](./reviews/3/index) +* - Reviewer + - Melissa Linkert, Sebastién Besson, Chris Allan, Jason Swedlow + - glencoesoftware + - Glencoe Software + - 2024-08-05 + - [Accept](./reviews/1b/index) +``` ## Overview diff --git a/rfc/2/review_1.md b/rfc/2/reviews/1/index.md similarity index 100% rename from rfc/2/review_1.md rename to rfc/2/reviews/1/index.md diff --git a/rfc/2/review_1b.md b/rfc/2/reviews/1b/index.md similarity index 100% rename from rfc/2/review_1b.md rename to rfc/2/reviews/1b/index.md diff --git a/rfc/2/review_2.md b/rfc/2/reviews/2/index.md similarity index 100% rename from rfc/2/review_2.md rename to rfc/2/reviews/2/index.md diff --git a/rfc/2/review_3.md b/rfc/2/reviews/3/index.md similarity index 100% rename from rfc/2/review_3.md rename to rfc/2/reviews/3/index.md From 93c701af6a6114cddc42773337e2b973fa74db3e Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 11 Sep 2024 09:56:46 +0200 Subject: [PATCH 2/9] Add Jeremy's accept --- rfc/2/reviews/3b/index.md | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 rfc/2/reviews/3b/index.md diff --git a/rfc/2/reviews/3b/index.md b/rfc/2/reviews/3b/index.md new file mode 100644 index 00000000..096cf964 --- /dev/null +++ b/rfc/2/reviews/3b/index.md @@ -0,0 +1,9 @@ +# Review 3 (update) + +## Review authors + +This review was submitted by Jeremy Maitin-Shepard (Google) via email. + +## Recommendation + +"Accept" From 8635beae7ada063e5c8da6d7178385fff8e3f442 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 11 Sep 2024 09:58:31 +0200 Subject: [PATCH 3/9] Update status list --- rfc/2/index.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/rfc/2/index.md b/rfc/2/index.md index 49b14f32..eb485ec2 100644 --- a/rfc/2/index.md +++ b/rfc/2/index.md @@ -128,6 +128,12 @@ This RFC is currently in RFC state (R6). - Glencoe Software - 2024-08-05 - [Accept](./reviews/1b/index) +* - Reviewer + - Jeremy Maitin-Shepard + - jbms + - Google + - 2024-09-11 + - [Accept](./reviews/3b/index) ``` ## Overview From 5ab115d604f32a6def13769a5674b30edbc9caea Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 11 Sep 2024 10:00:42 +0200 Subject: [PATCH 4/9] Correct typo --- rfc/1/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rfc/1/index.md b/rfc/1/index.md index dee12599..060ef9f1 100644 --- a/rfc/1/index.md +++ b/rfc/1/index.md @@ -34,19 +34,19 @@ This RFC is currently being responded to (R4). - [German BioImaging, e.V.](https://ror.org/05tpnw772) - 2023-12-23 - Author ([PR](https://github.com/ome/ngff/pull/222)) -* - Reviwer +* - Reviewer - Davis Bennett, John Bogovic, Michael Innerberger, Mark Kittisopikul, Virginia Scarlett, Yurii Zubov - d-v-b, bogovicj, minnerbe, mkitti, virginiascarlett, yuriyzubov - Janelia - 2024-02-26 - [Review](./reviews/2/index) -* - Reviwer +* - Reviewer - Kevin Yamauchi, Virginie Uhlmann - kevinyamauchi, vuhlmann - ETH, BiovisionCenter - 2024-03-05 - [Review](./reviews/1/index) -* - Reviwer +* - Reviewer - Matthew Hartley - mrmh2 - EMBL-EBI From 0c708f6808cf6919879f95a876297a38923059ed Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 11 Sep 2024 10:12:37 +0200 Subject: [PATCH 5/9] Correct name spelling --- rfc/2/index.md | 4 ++-- rfc/2/reviews/1/index.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/rfc/2/index.md b/rfc/2/index.md index eb485ec2..3ab73533 100644 --- a/rfc/2/index.md +++ b/rfc/2/index.md @@ -105,7 +105,7 @@ This RFC is currently in RFC state (R6). - 2024-03-14 - [Endorse](https://github.com/ome/ngff/pull/227#issuecomment-1998594492) * - Reviewer - - Melissa Linkert, Sebastién Besson, Chris Allan, Jason Swedlow + - Melissa Linkert, Sébastien Besson, Chris Allan, Jason Swedlow - glencoesoftware - Glencoe Software - 2024-05-23 @@ -123,7 +123,7 @@ This RFC is currently in RFC state (R6). - 2024-04-30 - [Review](./reviews/3/index) * - Reviewer - - Melissa Linkert, Sebastién Besson, Chris Allan, Jason Swedlow + - Melissa Linkert, Sébastien Besson, Chris Allan, Jason Swedlow - glencoesoftware - Glencoe Software - 2024-08-05 diff --git a/rfc/2/reviews/1/index.md b/rfc/2/reviews/1/index.md index d35d1921..99eb1277 100644 --- a/rfc/2/reviews/1/index.md +++ b/rfc/2/reviews/1/index.md @@ -3,7 +3,7 @@ ## Review authors This review was written by the following Glencoe Software team members: -- Sebastién Besson +- Sébastien Besson - Chris Allan - Marc Bruce - Jason Swedlow From 311e80664be461fe06081332f5ea264d058fcd2e Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Wed, 11 Sep 2024 14:52:21 +0200 Subject: [PATCH 6/9] Fix anchor links --- conf.py | 2 +- rfc/2/reviews/1/index.md | 18 +++++++++--------- rfc/2/reviews/1b/index.md | 2 +- 3 files changed, 11 insertions(+), 11 deletions(-) diff --git a/conf.py b/conf.py index 83ace581..5e414675 100644 --- a/conf.py +++ b/conf.py @@ -15,7 +15,7 @@ extensions = ["myst_parser"] source_suffix = [".rst", ".md"] -myst_heading_anchors = 4 +myst_heading_anchors = 5 templates_path = ['_templates'] exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', '.git', '.pytest_cache', '**/.pytest_cache', '**/.tox', 'README.md', 'LICENSE.md', 'CONTRIBUTING.md'] diff --git a/rfc/2/reviews/1/index.md b/rfc/2/reviews/1/index.md index 99eb1277..1c23db3b 100644 --- a/rfc/2/reviews/1/index.md +++ b/rfc/2/reviews/1/index.md @@ -11,7 +11,7 @@ This review was written by the following Glencoe Software team members: ## Summary -Overall, [this RFC](https://ngff.openmicroscopy.org/rfc/2/index.html) has merit, and the ability to use Zarr v3 features in NGFF would be generally useful. Sharding and additional codecs in particular are features we plan to make use of throughout the NGFF implementations we maintain. Our implementations are primarily in Java and Python, and include: +Overall, [this RFC](../../index) has merit, and the ability to use Zarr v3 features in NGFF would be generally useful. Sharding and additional codecs in particular are features we plan to make use of throughout the NGFF implementations we maintain. Our implementations are primarily in Java and Python, and include: * Tools to convert to and from NGFF (including but not limited to [bioformats2raw](https://github.com/glencoesoftware/bioformats2raw)) * Support for reading NGFF datasets in [OMERO](https://www.openmicroscopy.org/omero/) ([omero-zarr-pixel-buffer](https://github.com/glencoesoftware/omero-zarr-pixel-buffer)) @@ -20,21 +20,21 @@ However, we have a few high-level concerns that prevent a recommendation of "Acc ## Concern #1 - Limited Zarr v3 implementations -The [Background](https://ngff.openmicroscopy.org/rfc/2/index.html#background) section notes several libraries which implement Zarr v3. Several of these implementations claim to be limited experimental or prototype implementations (in particular, [zarrita](https://github.com/scalableminds/zarrita) and [zarr3-rs](https://github.com/clbarnes/zarr3-rs)). The only Java implementation we are aware of ([zarr-java](https://github.com/zarr-developers/zarr-java)) is not named in this RFC, but is similarly described in its README. The [state](https://zarr.readthedocs.io/en/stable/api/v3.html) of Zarr v3 support in the zarr-python reference implementation is also not mentioned. +The [Background](../../index.md#background) section notes several libraries which implement Zarr v3. Several of these implementations claim to be limited experimental or prototype implementations (in particular, [zarrita](https://github.com/scalableminds/zarrita) and [zarr3-rs](https://github.com/clbarnes/zarr3-rs)). The only Java implementation we are aware of ([zarr-java](https://github.com/zarr-developers/zarr-java)) is not named in this RFC, but is similarly described in its README. The [state](https://zarr.readthedocs.io/en/stable/api/v3.html) of Zarr v3 support in the zarr-python reference implementation is also not mentioned. -The [Implementation](https://ngff.openmicroscopy.org/rfc/2/index.html#implementation) section states that OME-Zarr implementers can take advantage of Zarr v3 support in existing libraries. It isn't clear how well that currently applies across a diverse set of languages and use cases (reading, writing, local storage vs cloud storage). If Zarr v3 support is not yet fully implemented in existing libraries, this will present a substantial challenge for OME-Zarr implementers. +The [Implementation](../../index.md#implementation) section states that OME-Zarr implementers can take advantage of Zarr v3 support in existing libraries. It isn't clear how well that currently applies across a diverse set of languages and use cases (reading, writing, local storage vs cloud storage). If Zarr v3 support is not yet fully implemented in existing libraries, this will present a substantial challenge for OME-Zarr implementers. -The [Implementation](https://ngff.openmicroscopy.org/rfc/2/index.html#implementation) section also includes a TODO to provide a reference implementation. The intention of this TODO is unclear - does this refer to a reference implementation of Zarr v3, or of NGFF backed by Zarr v3? In either case, does it indicate that a reference implementation exists and a link just needs to be added? Or does this suggest that no suitable reference implementation currently exists? +The [Implementation](../../index.md#implementation) section also includes a TODO to provide a reference implementation. The intention of this TODO is unclear - does this refer to a reference implementation of Zarr v3, or of NGFF backed by Zarr v3? In either case, does it indicate that a reference implementation exists and a link just needs to be added? Or does this suggest that no suitable reference implementation currently exists? ### Proposed resolution to concern #1 * Indicate which existing libraries have "production ready" support for Zarr v3. OME-Zarr implementers will need to know whether they can in fact use an existing library, or whether they should plan to implement underlying Zarr v3 support. - "Production ready" in this case means working write and read functionality in a versioned release. -* Replace the TODO in the [Implementation](https://ngff.openmicroscopy.org/rfc/2/index.html#implementation) section with information about existing reference implementations. +* Replace the TODO in the [Implementation](../../index.md#implementation) section with information about existing reference implementations. ## Concern #2 - Possibly conflicting statements on mixed metadata versions -The first paragraph of the [Proposal](https://ngff.openmicroscopy.org/rfc/2/index.html#proposal) section states that "Images that use the new version of OME-Zarr metadata MUST NOT use Zarr version 2 any more." This is seemingly at odds with the statement later in the same section that "Zarr version 2 and 3 metadata could even live side-by-side in the same hierarchy." If both metadata versions can exist within the same hierarchy, this raises several issues: +The first paragraph of the [Proposal](../../index.md#proposal) section states that "Images that use the new version of OME-Zarr metadata MUST NOT use Zarr version 2 any more." This is seemingly at odds with the statement later in the same section that "Zarr version 2 and 3 metadata could even live side-by-side in the same hierarchy." If both metadata versions can exist within the same hierarchy, this raises several issues: * If there are mismatches between the two sets of metadata, which one should be treated as authoritative? * If Zarr v3 features are used (e.g. sharding), what does v2 metadata look like? @@ -47,7 +47,7 @@ Either clarify that NGFF 0.5/Zarr v3 metadata MUST NOT be mixed with NGFF 0.4/Za ## Concern #3 - Proposal to upgrade 0.4 data -The [Proposal](https://ngff.openmicroscopy.org/rfc/2/index.html#proposal) section indicates both that 0.5+ data should be prioritized to reduce the burden on OME-Zarr implementers, and that users could "upgrade" their 0.4 data to 0.5. While technically feasible and well within the capabilities of the primary audience of the specification documents, this appears to place much of the burden on transition from 0.4 to 0.5 on end users. In particular, this may be problematic or even impossible for users with lower technical knowledge and for anyone using OME-Zarr in a read-only or large archive repository (for example [IDR](https://idr.openmicroscopy.org/)). While not all are publicly accessible, we are aware of academic and commercial use of NGFF 0.4 on the scale of hundreds of terabytes. +The [Proposal](../../index.md#proposal) section indicates both that 0.5+ data should be prioritized to reduce the burden on OME-Zarr implementers, and that users could "upgrade" their 0.4 data to 0.5. While technically feasible and well within the capabilities of the primary audience of the specification documents, this appears to place much of the burden on transition from 0.4 to 0.5 on end users. In particular, this may be problematic or even impossible for users with lower technical knowledge and for anyone using OME-Zarr in a read-only or large archive repository (for example [IDR](https://idr.openmicroscopy.org/)). While not all are publicly accessible, we are aware of academic and commercial use of NGFF 0.4 on the scale of hundreds of terabytes. In addition, the recommendation to upgrade data instead of promoting backwards compatibility is at odds with prevailing standards in the wider imaging community. This is also problematic considering the format's primary competitors in biomedical imaging (DICOM, HDF5) have maintained strict backwards compatibility policies for over two decades. We are not aware of any established precedent for requiring or requesting an end user to modify their data in this way, either in commercial image file formats or existing open file formats (e.g. [OME-TIFF](https://ome-model.readthedocs.io/en/stable/ome-tiff/)). We would anticipate significant pushback from the NGFF user community in this case. Setting the precedent here that data must be upgraded also implies that any future specification version may contain the same requirement. @@ -55,11 +55,11 @@ In addition, the recommendation to upgrade data instead of promoting backwards c * Add that OME-Zarr implementers MUST clearly indicate which released version(s) of their implementation(s) support NGFF 0.4, which support NGFF 0.5, and which (if any) support a potentially transitional mix of 0.4 and 0.5. * Clarify whether newly written NGFF 0.5/Zarr v3 data MAY or MUST NOT contain NGFF 0.4/Zarr v2 metadata to improve compatibility. -* Rephrase third paragraph of [Proposal](https://ngff.openmicroscopy.org/rfc/2/index.html#proposal) section to indicate that implementers SHOULD consider handling both NGFF 0.4 and NGFF 0.5. It should also be made clear that implementers CANNOT assume that all existing NGFF 0.4 can or will be upgraded. +* Rephrase third paragraph of [Proposal](../../index.md#proposal) section to indicate that implementers SHOULD consider handling both NGFF 0.4 and NGFF 0.5. It should also be made clear that implementers CANNOT assume that all existing NGFF 0.4 can or will be upgraded. ## Concern #4: Metadata namespaces -The [Changes to the OME-Zarr metadata](https://ngff.openmicroscopy.org/rfc/2/index.html#changes-to-the-ome-zarr-metadata) section proposes to introduce a dedicated key in the Zarr array or group attributes for all metadata related to the OME-NGFF specification. There are two separate concerns about the proposed usage of the specification URL as this dedicated key. +The [Changes to the OME-Zarr metadata](../../index.md#changes-to-the-ome-zarr-metadata) section proposes to introduce a dedicated key in the Zarr array or group attributes for all metadata related to the OME-NGFF specification. There are two separate concerns about the proposed usage of the specification URL as this dedicated key. Firstly, whilst the JSON specification does not to our knowledge impose any restriction on strings, in our experience, URLs are more often found as values rather than keys. Secondly, the usage of version specific keys rather than a top-level named key e.g. `ome` introduces a precedent for every following version of the OME-NGFF specification which will have to adopt this convention. For OME-Zarr datasets, it also means a Zarr array or group could include multiple name/value pairs associated with different versions of the OME-NGFF specification. This raises the same issues as discussed in Concern#2 about resolving mismatches and precedence. diff --git a/rfc/2/reviews/1b/index.md b/rfc/2/reviews/1b/index.md index 7b1feaca..54731de4 100644 --- a/rfc/2/reviews/1b/index.md +++ b/rfc/2/reviews/1b/index.md @@ -10,7 +10,7 @@ This review was written by the following Glencoe Software team members: The changes to RFC-2 in https://github.com/ome/ngff/pull/250 are sufficient to update our recommendation from "Major changes" to "Accept". -Our original [Concern #1](https://ngff.openmicroscopy.org/rfc/2/review_1.html#concern-1-limited-zarr-v3-implementations) is addressed by the addition of references to zarr-python and zarr-java, as well as [clarifications to the zarr-java API](https://github.com/zarr-developers/zarr-java/issues/5) prompted by work on [v2/v3 conversion tools](https://github.com/glencoesoftware/zarr2zarr). Remaining concerns in [our original review](https://ngff.openmicroscopy.org/rfc/2/review_1.html) are addressed by stronger language around how applications should handle multiple versions and the usage of ‘ome’ as the top-level key for the OME-NGFF metadata in zarr.json. +Our original [Concern #1](../1/index.md#concern-1-limited-zarr-v3-implementations) is addressed by the addition of references to zarr-python and zarr-java, as well as [clarifications to the zarr-java API](https://github.com/zarr-developers/zarr-java/issues/5) prompted by work on [v2/v3 conversion tools](https://github.com/glencoesoftware/zarr2zarr). Remaining concerns in [our original review](../1/index) are addressed by stronger language around how applications should handle multiple versions and the usage of ‘ome’ as the top-level key for the OME-NGFF metadata in zarr.json. ## Recommendation From 5c25ae75aecc70ce7ed37596e35a25779759ce1f Mon Sep 17 00:00:00 2001 From: Yaroslav Halchenko Date: Wed, 11 Sep 2024 14:30:37 -0400 Subject: [PATCH 7/9] Add my re-review (#2) of RFC-2 --- rfc/2/index.md | 6 ++++++ rfc/2/reviews/2b/index.md | 25 +++++++++++++++++++++++++ 2 files changed, 31 insertions(+) create mode 100644 rfc/2/reviews/2b/index.md diff --git a/rfc/2/index.md b/rfc/2/index.md index 3ab73533..2fc97417 100644 --- a/rfc/2/index.md +++ b/rfc/2/index.md @@ -134,6 +134,12 @@ This RFC is currently in RFC state (R6). - Google - 2024-09-11 - [Accept](./reviews/3b/index) +* - Reviewer + - Yaroslav O. Halchenko + - yarikoptic + - Dartmouth College, DANDI Project + - 2024-09-11 + - [Accept](./reviews/2b/index) ``` ## Overview diff --git a/rfc/2/reviews/2b/index.md b/rfc/2/reviews/2b/index.md new file mode 100644 index 00000000..a30f411e --- /dev/null +++ b/rfc/2/reviews/2b/index.md @@ -0,0 +1,25 @@ +# Review 2 (update) + +## Review authors +This review was written by: +- Yaroslav O. Halchenko (Dartmouth College, DANDI Project) + +## Summary + +Comments from the initial round of reviews were proposed, accepted (with minor tune ups after) in https://github.com/ome/ngff/pull/250 . The changes to RFC-2 in that PR are sufficient to update our recommendation from "Major changes" to "Accept". + +- Concern on compatibility with Zarr v2/OME 0.4 was largely addressed: it is now RECOMMENDED to maintain compatibility. +- Concern on v2 and v3 co-existence was addressed by more explicit formulation that it is possible but NOT RECOMMENDED [^1], and that 0.5 SHOULD be treated preferentially in case of both v2 and v3 metadata co-existence. +- Recommendation on separation of the "name" of the standard from the versioning was addressed by introduction of `ome` as the top-level key for the OME-NGFF metadata in zarr.json with a `version` key within that `ome` record. + - unfortunately the concept of "namespaces" is not yet formalized in Zarr itself, WiP in [ZEP004: Namespaces](https://github.com/zarr-developers/zarr-specs/pull/262/files). + So it would remain possible that changes in the future would be needed to align with Zarr's formalization of namespaces. +- My questioning of the value to bundle together support of Zarr v3 and changes to metadata was largely addressed by addition to "Drawbacks, risks, alternatives, and unknowns": even without formalization of "Zarr namespaces" it would be useful to have `ome` as the top-level key for the OME-NGFF metadata in zarr.json. + +[^1]: according to [RFC 2119: Section 4](https://datatracker.ietf.org/doc/html/rfc2119#section-4), even though "NOT RECOMMENDED" is not explicitly listed among the list of key words, it is to be treated similarly to "SHOULD NOT", so we are fine here. + +Remaining concerns in the original review are addressed by stronger language around how applications should handle multiple versions and the usage of ‘ome’ as the top-level key for the OME-NGFF metadata in zarr.json. + + +## Recommendation + +"Accept". From c92b2fee09909505ad2b6fe118544276d38c7d73 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Fri, 13 Sep 2024 10:53:54 +0200 Subject: [PATCH 8/9] Add RFC navigation elements --- rfc/2/comments/index.md | 9 +++++++++ rfc/2/responses/index.md | 9 +++++++++ rfc/2/reviews/index.md | 9 +++++++++ rfc/2/versions/index.md | 9 +++++++++ 4 files changed, 36 insertions(+) create mode 100644 rfc/2/comments/index.md create mode 100644 rfc/2/responses/index.md create mode 100644 rfc/2/reviews/index.md create mode 100644 rfc/2/versions/index.md diff --git a/rfc/2/comments/index.md b/rfc/2/comments/index.md new file mode 100644 index 00000000..39447631 --- /dev/null +++ b/rfc/2/comments/index.md @@ -0,0 +1,9 @@ +# Comments + +Additional comments of RFC-2: + +```{toctree} +:maxdepth: 1 +:glob: +*/index +``` diff --git a/rfc/2/responses/index.md b/rfc/2/responses/index.md new file mode 100644 index 00000000..0d478f55 --- /dev/null +++ b/rfc/2/responses/index.md @@ -0,0 +1,9 @@ +# Responses + +Responses from the authors of RFC-2: + +```{toctree} +:maxdepth: 1 +:glob: +*/index +``` diff --git a/rfc/2/reviews/index.md b/rfc/2/reviews/index.md new file mode 100644 index 00000000..69a8d26b --- /dev/null +++ b/rfc/2/reviews/index.md @@ -0,0 +1,9 @@ +# Reviews + +Reviews of RFC-2: + +```{toctree} +:maxdepth: 1 +:glob: +*/index +``` diff --git a/rfc/2/versions/index.md b/rfc/2/versions/index.md new file mode 100644 index 00000000..3be0622c --- /dev/null +++ b/rfc/2/versions/index.md @@ -0,0 +1,9 @@ +# Versions + +Key versions of the RFC which have been sent for review, etc. + +```{toctree} +:maxdepth: 1 +:glob: +*/index +``` From fbeba6ad21368783d5f915b35c5c456af378c194 Mon Sep 17 00:00:00 2001 From: Josh Moore Date: Fri, 13 Sep 2024 11:31:03 +0200 Subject: [PATCH 9/9] RFC-2: state S1 --- rfc/2/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfc/2/index.md b/rfc/2/index.md index 2fc97417..5efb793b 100644 --- a/rfc/2/index.md +++ b/rfc/2/index.md @@ -13,7 +13,7 @@ Adopt the version 3 of Zarr for OME-Zarr. ## Status -This RFC is currently in RFC state (R6). +This RFC is currently in RFC state (R2). ```{list-table} Record :widths: 8, 20, 20, 20, 15, 10