Open MoBIE projects from data other than a MoBIE project

[k-dominik]

Dear @tischi,

mobie looks so awesome! Getting your data in is still a little clunky though (as far as I know) with having to create a project first and all. I think many use cases could go directly from memory, ideally from fiji to open a mobie viewer instance with a given image, label-image and table. We have a PR open for the ilastik fiji plugin, that allows running object classification with a pre-trained project and that would load both the label image and a corresponding feature table back. Ideally one would be able now to easily open mobie - no writing of temp files.

Cheers :)
Dominik

[tischi]

Getting your data in is still a little clunky though (as far as I know) with having to create a project first and all.

The vision here is that in ilastik there should be just a "Save as OME-Zarr" and that should contain all the information. We don't have tables yet though, but you could already support saving the image and label masks.

ideally from fiji to open a mobie viewer instance with a given image, label-image and table

Yes, I agree! We should support this! We are lacking person-power for the development at the moment. Do you happen to have a good Java programmer at hand?

[constantinpape]

The vision here is that in ilastik there should be just a "Save as OME-Zarr" and that should contain all the information. We don't have tables yet though, but you could already support saving the image and label masks.

I agree, the best option would be to save everything as ome.zarr to avoid in memory data transfer logic (and then either also generate the json file for MoBIE to understand the data layout, or we simplify this part that we can open a MoBIE project from the ome.zarr directly).

For the table: while we don't support tables in ome.zarr yet, we could also use tsv files in the meantime, which should be easy to export from the ilastik side.

[tischi]

or we simplify this part that we can open a MoBIE project from the ome.zarr directly

Definitely something we want to do!

[tischi]

We can collect some use-cases:

MorpholibJ Fiji

• Fiji UI is open
• An IntensityImagePlus is open
• A LabelMaskImagePlus is open
• A MorpholibJ ResultsTable is open
• View this as a MoBIE project

[tischi]

Branch to work on this: https://github.com/mobie/mobie-viewer-fiji/tree/issue811

[constantinpape]

Opening images in MoBIE from the command line

It would also be useful to be able to open images in MoBIE from the command line. For me an example application would be visualizing features associated with segmented objects, very similar to the MorpholibJ use-case from above. But instead the feature computation etc. would be in python, so it would be nice to have a convenient way to open MoBIE directly from it.
The workflow would be something like this:

• Perform segmentation and feature computation in python.
• Save image, segmentation and features temporarily in a MoBIE compatible image format
• Open in MoBIE via command line interface (called as a python subprocess)

For this the CLI should support (at least) the following arguments:

$ run-mobie -i image1.tif image2.tif -s segmentation1.tif segmentation2.tif -t table1.tsv table2.tsv

So after -i would be a list of image files , after -s the list of segmentations and after -t the list of tables. (All lists can ofc just have a single following argument; segmentations and tables optional).

[tischi]

Thank you @constantinpape! That is in fact exactly how I also imagined the command line call to look like.

@kutra would that work for your use-cases as well?

[k-dominik]

Hey @tischi, my handle is @k-dominik :P ;_)

I would also really much like to have the ability to open mobie with appropriate data from the command line, pretty much what @constantinpape described. I think what's a bit missing is how to specify the "flavor" of table you add to it. Depending on the source, column names might vary and so on. So of course there could be a small GUI that would pop up allow you to do these choices. On the other hand, the sources of such tables that I can think of are currently a handful (morpholibj, analyze particles, regionprops, ilastik) and it would be great to support those with some form of magic :).

side note:

So after -i would be a list of image files , after -s the list of segmentations and after -t the list of tables.

I am not a fan of allowing multiple arguments per option - There might be use-cases like having stacks across multiple files and that can get messy then really quickly. I'm more a fan of specifying options multiple times... Or to phrase it differently, multiple arguments should maybe mean stacks.

$ run-mobie -i image1.tif -i image2.tif ...

also related to allowing multiple images - would you allow multiple tables from different sources for a single image. Or different table sources for different images? Is the use case with multiple input images so important?

[constantinpape]

On the other hand, the sources of such tables that I can think of are currently a handful (morpholibj, analyze particles, regionprops, ilastik) and it would be great to support those with some form of magic :).

I think @tischi is working on code to handle this (basically check the column names in the table and match them to the internal MoBIE representations).

I am not a fan of allowing multiple arguments per option - There might be use-cases like having stacks across multiple files and that can get messy then really quickly.

I don't have a strong opinion on this and from my end both -i image1 image2 or -i image1 -i image2 would be fine.

would you allow multiple tables from different sources for a single image

To be nitpicky: in MoBIE syntax we would refer to tables associated with a segmentation, an image in the MoBIE definition is a pure intensity image and does not support tables.

would you allow multiple tables from different sources for a single image. Or different table sources for different images?

My suggestion would be to stick to the simplest solution that supports the relevant use-case. That is: each segmentation can have a single table associated with it, and we match tables to segmentation by the order of the cli argument, so:

run-mobie [...] -s segmentation1 -s segmentation2 -t table_seg1 -t table_seg2

Is the use case with multiple input images so important?

The case of supporting multiple images and segmentations: yes, it's important, I think only supporting to add a single image and/or segmentation via the CLI would be very limiting.
The case of supporting multiple tables per segmentation: it's not so important, since the tables can simply be added via the table GUI.

[tischi]

@k-dominik what image and table file formats would you like to use?

Column names

We could have a --table_format with, e.g.morpholibj or mobie, which would specify the the column name mapping. Not sure about the actual file format here. Maybe a long as it is csv or tsv it is fine?

For custom tables where we don't have a profile I see two options: provide a config file -c config.json?txt? or have several arguments such as -acx for anchor column x.

What do you think?

Image file formats

I think most file formats are obvious from the file ending. I think the most tricky case may be ilastik h5 files, because they are not supported by Bio-Formats, so we have to add an extra dependency and also somehow know that this is an ilastik h5 and not some other h5.

What about adding a parameter --image_format? It would be good if we could use these Strings as that is what we use in mobie-io.

[constantinpape]

I think the most tricky case may be ilastik h5 files, because they are not supported by Bio-Formats, so we have to add an extra dependency and also somehow know that this is an ilastik h5 and not some other h5.

You won't need an extra dependency for ilastik outputs besides hdf5, they are not really a separate file format, but I guess you would need to write a loader for them in mobie-io that supports single scale hdf5 files.

However, the issue is that the output is not fully standardized (both in terms of the axis that are saved and the internal paths of the dataset in the h5 root).
Maybe @k-dominik could specify the kind of ilastik output for which he'd want to use it. (Is it always 5D TCZYX data with internal path exported_data?)

What about adding a parameter --image_format? It would be good if we could use these Strings as that is what we use in mobie-io.

Do you want to make it mandatory or optional? (And in the optional case try to figure it out from the file extension.)

[k-dominik]

Maybe @k-dominik could specify the kind of ilastik output for which he'd want to use it. (Is it always 5D TCZYX data with internal path exported_data?)

Currently it would be an nd-array with an arbitrary dataset name (it's up to the user to specify). Dimensionality is also not enforced or anything, but axis labels are always exported.

Example java code for loading ilastik hdf5 files is in our ilastik4ij repo. Maybe we could restructure it to make it reusable more easily in the context of mobie?

What probably makes the hdf5 datasets from ilastik identifiable (this would be infinitely backwards compatible, since this is added since forever) would be the serialized vigra axistags in the axistags attribute.

For the table files, we offer options for .csv and .h5 tables, so supporting only csv is not a problem I'd say.

[tischi]

@k-dominik I made a new issue for the ilastik image files.

Regarding --image_format: I guess it is nice if it is optional?!

@constantinpape what about I add a static method in mobie-io to ImageDataFormat with the API:
ImageDataFormat formatFromPath( String imagePath ) and we see how much we can auto-detect?

[constantinpape]

Regarding --image_format: I guess it is nice if it is optional?!

@constantinpape what about I add a static method in mobie-io to ImageDataFormat with the API: ImageDataFormat formatFromPath( String imagePath ) and we see how much we can auto-detect?

Sounds good.

[tischi]

I merge the branch issue811 in which I was working on this issue into develop.
Otherwise it would have gotten too complex for me keeping track of all the changes in the different branches.
This is also what I uploaded just now to MoBIE-beta.
I tried the platybrowser and the covid-if projects and I don't think I broke anything....