Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

investigate dropping ome-zarr dependency #123

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

investigate dropping ome-zarr dependency #123

wants to merge 2 commits into from

Conversation

will-moore
Copy link
Member

@will-moore will-moore commented Dec 13, 2024 •
edited
Loading

Investigating a lighter-weight alternative to ome-zarr-py.

Work in progress...

Not looking to fully replace ome-zarr-py yet, but just investigating what an alternative could look like to inform discussions.

Pros:

  • Mapping from OME-Zarr metadata to napari data happens in one step, instead of being split between ome-zarr-py and napari-ome-zarr.

Currently handles channels metadata and labels. Testing with:

$ napari --plugin napari-ome-zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0062A/6001240_labels.zarr

$ napari --plugin napari-ome-zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.5/idr0066/ExpD_chicken_embryo_MIP.ome.zarr

$ napari --plugin napari-ome-zarr https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846152.zarr

@will-moore
Copy link
Member Author

As discussed with @joshmoore this morning, I looked into whether https://github.com/BioImageTools/ome-zarr-models-py could perform some of the graph traversal logic in this PR, e.g. multiscales -> labels or bioformats2raw -> multiscales (not yet done in this PR). But I don't see any of that functionality in ome-zarr-models.py?
cc @dstansby

@dstansby
Copy link
Contributor

graph traversal logic in this PR

I had a quick look at the diff, but didn't quite understand what the required logic is. Could you explain a bit more? (or maybe add some short docstrings to the new classes/methods?)

We are generally 👍 on adding helpful functionality to ome-zarr-models-py, and I'm going to be sprinting to a first release tomorrow actually, so now is a good time to request stuff 😄

will-moore
will-moore commented Dec 17, 2024
View reviewed changes
napari_ome_zarr/ome_zarr_reader.py
def matches(group: Group) -> bool:
return "multiscales" in Spec.get_attrs(group)

def children(self):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dstansby By "graph traversal" logic, I mean, if I start with multiscales group e.g. group = zarr.open("https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr") I then want to get the labels (if they exist). Here this is implemented in the children() method, where we know to look in a child "labels" group and check attrs for "labels": ["labels1.zarr", "labels2.zarr"] then return objects for those child labels so that the arrays (and metadata) can be added to the layers that are passed to napari.

I don't see that ome-zarr-models-py includes that kind of logic for traversing the graph between these objects?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can get the list of labels paths from Image.attributes.labels. But the labels part of the spec just says these point to "labels objects", which I don't think are more specifically defined anywhere else?

If the OME-Zarr spec was more prescriptive about what these "labels objects" were (are they meant to be groups with image-label metadata ??) then we could certainly do more, but I don't think the spec allows us to make those assumptions unfortunately.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am only now reading version 0.5 of OME-zarr, and see that the labels section is much improved over 0.4 😄 . It's definitely within scope of ome-zarr-models-py to provide logic for getting from an Image dataset to the labels dataset if it's in the metadata. Tracking issue at BioImageTools/ome-zarr-models-py#92

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently in the tutorial at https://github.com/BioImageTools/ome-zarr-models-py/blob/main/docs/tutorial.py
If I add:

print(ome_zarr_image.attributes.labels)

I get None (even though that image does have labels).
I don't see any population of the labels in https://github.com/BioImageTools/ome-zarr-models-py/blob/7659a114a2428fe9d8acbd06aa7bc1c9d32624bb/src/ome_zarr_models/v04/image.py#L85 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

even though that image does have labels

There's a labels group, but looking at that dataset in the validator the top level group is missing the labels metadata, which is why .labels is giving None.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The image group doesn't have labels metadata though, so ome-zarr-models-py has no way to find a way from the image group to the labels group.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be precise, if you look at the image group in the validator, it should have a "labels" key at the same level as the "multiscales" and "omero" keys. If that was there, the paths under the "labels" key would be in the .labels attribute in ome-zarr-models-py

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh hold on, am I just reading the spec wrong? Does:

The special group "labels" found under an image Zarr

Really mean:

The special Zarr group "labels" found inside an image Zarr group

?

If so then we should definitely implement that in ome-zarr-models-py!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, image.zarr/labels/ group.
This is shown a bit more clearly in the layout at https://ngff.openmicroscopy.org/0.4/index.html#image-layout

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, I always thought the "labels" group was an arbitrary name and the example was just an example 🤦 - thanks for explaining, and I'll let you know once I've implemented this in ome-zarr-models-py 😄

@will-moore
Copy link
Member Author

One issue I'm having with supporting bioformats2raw.layout here is how to load the /OME/METADATA.ome.xml (which could be local or remote etc).

E.g. https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846151.zarr/OME/METADATA.ome.xml

I'm trying something like:

from zarr.core.buffer import default_buffer_prototype
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846151.zarr"
group = zarr.open(url)
xml_data = await group.store.get("OME/METADATA.ome.xml", prototype=default_buffer_prototype())

but I want to avoid using async await if I can.

In https://github.com/ome/ome-zarr-py/pull/174/files it looks like this only handles local OME.xml files with root = ET.parse(filename)

@joshmoore
Copy link
Member

In ome/ome-zarr-py#174 (files) it looks like this only handles local OME.xml files with root = ET.parse(filename)

Then that would have been a bug. I assume a method like get_json() (get_text or get_contents) would have been needed to slurp the XML.

@will-moore
Copy link
Member Author

Seems this works for getting XML, but need to check if there's a better way that doesn't use testing classes...

import zarr
from zarr.core.buffer import default_buffer_prototype
from zarr.testing.stateful import SyncStoreWrapper
url = "https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0048A/9846151.zarr"
group = zarr.open(url)
store = group.store
wrapper = SyncStoreWrapper(store)
xml_data = wrapper.get("OME/METADATA.ome.xml", prototype=default_buffer_prototype())
print("xml_data", xml_data.to_bytes())

@joshmoore
Copy link
Member

Better to just use the SyncMixing for now: https://github.com/zarr-developers/zarr-python/blob/4455726a0dbdeebb00d146b3b7a4bfa4e63374b5/src/zarr/core/sync.py#L181

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dstansby dstansby dstansby left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@will-moore @dstansby @joshmoore