Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to new jzarr 0.4.2 #203

Merged
merged 7 commits into from
Nov 15, 2023

Conversation

melissalinkert
Copy link
Member

Prompted by ome/ZarrReader#54, see https://github.com/zarr-developers/jzarr.

Discussed briefly with @sbesson. I am not suggesting that we consider this for bioformats2raw 0.7.0, it's more a place to start. Whatever we decide to do here should likely be applied to other repos where we use jzarr (including but not limited to raw2ometiff).

@joshmoore
Copy link
Contributor

These are probably the most relevant missing changes:

+  [gs/master~6] Array nesting: store in .zarray if set
+  [gs/master~7] Array nesting: Handle empty zarr arrays
+  [gs/master~8] Array nesting: Add the ability to use N5-style nested layout

@melissalinkert melissalinkert changed the title Update to new jzarr 0.3.7 Update to new jzarr 0.4.0 Jun 28, 2023
Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following a discussion with @melissalinkert, a few comments:

  • the com.bc.zarr:jzarr library should be considered as inactive
  • the cleanup of all intermediate repositories and the ability to use Maven Central is really nice
  • looking nature of the changes between the 0.3.5 and the 0.4.0 changes bcdev/jzarr@0.3.5...zarr-developers:jzarr:0.4.0, I see nothing of particular concerns:
    • essential metadata got added to the POM like the license 👍
    • a few API additions which could be consumed by the converter e.g. the multi-threaded blosc compression
    • the bump of the jackson dependency which is overriden by a more recent versions from ome:formats-gpl anyways
  • at present, https://github.com/zarr-developers/jzarr feels like the best place to discuss & propose new API additions which would be relevant to the needs of this library
  • we will likely need to unify the change across all conversion utilities to avoid a mixture of jzarr

Could we update

// no getter for DimensionSeparator in ZarrArray
// check that the correct separator was used by checking
// that the expected first chunk file exists
assertTrue(output.resolve("0/0/0/0/0/0/0").toFile().exists());
to use the new dimension separator getter available in the new version?

@sbesson
Copy link
Member

sbesson commented Sep 23, 2023

👍 maybe let's just remove the outdated test comment and then this should be ready

   // no getter for DimensionSeparator in ZarrArray
    // check that the correct separator was used by checking
    // that the expected first chunk file exists

Next question is whether we want to plan a 0.7.0 release or a release candidate that can be consumed in all the related converter utility PRs adjusting the dependency to use dev.zarr

@melissalinkert
Copy link
Member Author

Test comment updated in 609f092. As discussed separately, this PR should not be merged right now, pending investigation of zarr-developers/jzarr#14.

build.gradle Outdated Show resolved Hide resolved
@sbesson sbesson self-requested a review October 31, 2023 21:18
Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retested the last commit and confirmed that:

  • the array metadata of the generated Zarr datasets no longer include the additional numThreads key
  • the Zarr datasets can be opened both using the zarr-python implementation and another Java implementation e.g. raw2ometiff

A few recent maintenance issues (zarr-developers/jzarr#17) are further reinforcing my feeling expressed in #203 (review) i.e. the inactive state of com.bc.zarr:jzarr effectively elevates the dev.zarr:jzarr fork as the new reference Java implementation for the Zarr v2 specification.

@sbesson sbesson changed the title Update to new jzarr 0.4.0 Update to new jzarr 0.4.2 Nov 3, 2023
@sbesson sbesson merged commit b21ed19 into glencoesoftware:master Nov 15, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants