Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix bug reading dragonfly acquisitions #215

Merged
merged 4 commits into from
Apr 9, 2024
Merged

Fix bug reading dragonfly acquisitions #215

merged 4 commits into from
Apr 9, 2024

Conversation

ieivanov
Copy link
Contributor

@ieivanov ieivanov commented Apr 2, 2024 •
edited
Loading

This PR fixes a bug in reading data acquired with the Dragonfly acquisition pipeline.

In the Dragonfly acquisition pipeline, we start with a large position list, but not all of these positions may be acquired. The resulting ome.tif dataset will have a large number of positions in the MM metadata, but fewer data series. In this PR, I limit the xdata size to the number of actual positions acquired. The site index is only accessible from the filename. There may be a better way to get the number of tif files and their filenames than doing a glob in the root directory.

@ieivanov ieivanov marked this pull request as ready for review April 3, 2024 20:17
@ieivanov ieivanov requested a review from ziw-liu April 3, 2024 20:17
@ziw-liu ziw-liu added bug Something isn't working μManager Micro-Manager files and metadata labels Apr 3, 2024
@ieivanov ieivanov merged commit fac2c13 into unified-api Apr 9, 2024
7 checks passed
@ieivanov ieivanov deleted the bugfix/read_dragonfly_acquisitions branch April 9, 2024 18:25
ieivanov added a commit that referenced this pull request Apr 15, 2024
@ieivanov
Squashed commit of the following:
3f00215 
commit fac2c13
Author: Ivan Ivanov <[email protected]>
Date:   Tue Apr 9 11:25:36 2024 -0700

    Fix bug reading dragonfly acquisitions (#215)

    * fix bug reading dragonfly acquisitions

    * fix typo

    * style

    * bugfix

commit 0c6984e
Author: Ivan Ivanov <[email protected]>
Date:   Mon Mar 11 12:35:51 2024 -0700

    Fix bug determining number of rows and cols (#214)

    * fix bug determining number of rows and cols

    * add another XY Stage variation

    * add docs and fix style

commit 3ab89ba
Author: Ziwen Liu <[email protected]>
Date:   Mon Mar 4 11:02:49 2024 -0800

    Universal API implementations for Micro-Manager OME-TIFF and NDTiff (#185)

    * wip: draft mmstack ome-tiff fov

    * MM FOV base class

    * tests

    * bump tifffile

    * comment

    * fix indent after rebase

    * use get default

    * test pixel indexing

    * set MM metadata

    * style

    * update dependencies

    * add xarray

    * move old readers to the `_deprecated` namespace

    * uapi for ndtiff

    * refactor test setup to parametrize by dataset
    use globals instead of fixtures since parametrization happens before fixture evaluation

    * convert mmstack

    * fix and test chunking

    * fix metadata conversion and test ndtiff

    * update cli

    * fix scaling

    * test 1.4 and incomplete ome-tiffs

    * move reader tests

    * deprecate reader tests

    * update deprecated tests

    * update ngff tests

    * isort

    * update black target to 3.10

    * lint

    * fix download paths

    * update docs references and theme

    * untrack autogenerated file

    * ignore execution time file

    * add github icon

    * update docstring

    * update docstring

    * show channel names and chunk size in info

    * print plate chunk size if verbose

    * fallback for pixel size

    * remove log level setting

    * do not filter logs and warnings in reader

    * avoid root logger

    * isort

    * set default logging level to INFO

    * format docstring

    * improve conversion messages

    * black

    * fix ome-tiff channel name indexing

    * fix ndtiff channel name indexing

    * update converter test

    * remove use of os.path in `reader`

    * expand _check_ndtiff checks

    * fix iteration

    * fix python 3.10

    using `Path.glob(*/)` to get subdirs was added in 3.11

    * bump zarr version to include resizing fix
    zarr-developers/zarr-python#1540

    * fix cli default

    * set log level with an environment variable

    * fix unset

    * catch non-existent page

    * implement fallback for incomplete channel names
    workaround for the Dragonfly microscope where the multi-camera setup only has one channel name written

    * add debug logs

    * handle virtual frames

    * try reading pages from TiffFile directly

    * filter error logs about ImageJ metadata being broken
    this is a known MM limitation when writing OME-TIFFs

    * fix regex

    * remove use of os.path in `convert.py`

    * better channel indexing in `_get_summary_metadata`

    * style

    * safer NoneType check

    * private default axis names for NDTiff

    * update documentation to reflect new entry point

    * add repr to MM FOV and dataset types

    * rename mm_meta and expose summary metadata

    * add MicroManagerFOVMapping.root

    * add MicroManagerFOVMapping.zyx_scale

    * add warning log for failed position grid

    * fix grid layout

    * suppress hypothesis flakiness

    * different health check suppression

    ---------

    Co-authored-by: Ivan Ivanov <[email protected]>
ieivanov added a commit that referenced this pull request Apr 15, 2024
@ieivanov @ziw-liu
Sort keys in metadata .json files (#211)
420f052 
* wip: draft mmstack ome-tiff fov

* MM FOV base class

* tests

* bump tifffile

* comment

* fix indent after rebase

* use get default

* test pixel indexing

* set MM metadata

* style

* update dependencies

* add xarray

* move old readers to the `_deprecated` namespace

* uapi for ndtiff

* refactor test setup to parametrize by dataset
use globals instead of fixtures since parametrization happens before fixture evaluation

* convert mmstack

* fix and test chunking

* fix metadata conversion and test ndtiff

* update cli

* fix scaling

* test 1.4 and incomplete ome-tiffs

* move reader tests

* deprecate reader tests

* update deprecated tests

* update ngff tests

* isort

* update black target to 3.10

* lint

* fix download paths

* update docs references and theme

* untrack autogenerated file

* ignore execution time file

* add github icon

* update docstring

* update docstring

* show channel names and chunk size in info

* print plate chunk size if verbose

* fallback for pixel size

* remove log level setting

* do not filter logs and warnings in reader

* avoid root logger

* isort

* set default logging level to INFO

* format docstring

* improve conversion messages

* black

* fix ome-tiff channel name indexing

* fix ndtiff channel name indexing

* update converter test

* remove use of os.path in `reader`

* expand _check_ndtiff checks

* fix iteration

* fix python 3.10

using `Path.glob(*/)` to get subdirs was added in 3.11

* bump zarr version to include resizing fix
zarr-developers/zarr-python#1540

* fix cli default

* set log level with an environment variable

* fix unset

* catch non-existent page

* implement fallback for incomplete channel names
workaround for the Dragonfly microscope where the multi-camera setup only has one channel name written

* add debug logs

* handle virtual frames

* try reading pages from TiffFile directly

* filter error logs about ImageJ metadata being broken
this is a known MM limitation when writing OME-TIFFs

* fix regex

* remove use of os.path in `convert.py`

* better channel indexing in `_get_summary_metadata`

* style

* safer NoneType check

* private default axis names for NDTiff

* sort metadata keys

* Squashed commit of the following:

commit fac2c13
Author: Ivan Ivanov <[email protected]>
Date:   Tue Apr 9 11:25:36 2024 -0700

    Fix bug reading dragonfly acquisitions (#215)

    * fix bug reading dragonfly acquisitions

    * fix typo

    * style

    * bugfix

commit 0c6984e
Author: Ivan Ivanov <[email protected]>
Date:   Mon Mar 11 12:35:51 2024 -0700

    Fix bug determining number of rows and cols (#214)

    * fix bug determining number of rows and cols

    * add another XY Stage variation

    * add docs and fix style

commit 3ab89ba
Author: Ziwen Liu <[email protected]>
Date:   Mon Mar 4 11:02:49 2024 -0800

    Universal API implementations for Micro-Manager OME-TIFF and NDTiff (#185)

    * wip: draft mmstack ome-tiff fov

    * MM FOV base class

    * tests

    * bump tifffile

    * comment

    * fix indent after rebase

    * use get default

    * test pixel indexing

    * set MM metadata

    * style

    * update dependencies

    * add xarray

    * move old readers to the `_deprecated` namespace

    * uapi for ndtiff

    * refactor test setup to parametrize by dataset
    use globals instead of fixtures since parametrization happens before fixture evaluation

    * convert mmstack

    * fix and test chunking

    * fix metadata conversion and test ndtiff

    * update cli

    * fix scaling

    * test 1.4 and incomplete ome-tiffs

    * move reader tests

    * deprecate reader tests

    * update deprecated tests

    * update ngff tests

    * isort

    * update black target to 3.10

    * lint

    * fix download paths

    * update docs references and theme

    * untrack autogenerated file

    * ignore execution time file

    * add github icon

    * update docstring

    * update docstring

    * show channel names and chunk size in info

    * print plate chunk size if verbose

    * fallback for pixel size

    * remove log level setting

    * do not filter logs and warnings in reader

    * avoid root logger

    * isort

    * set default logging level to INFO

    * format docstring

    * improve conversion messages

    * black

    * fix ome-tiff channel name indexing

    * fix ndtiff channel name indexing

    * update converter test

    * remove use of os.path in `reader`

    * expand _check_ndtiff checks

    * fix iteration

    * fix python 3.10

    using `Path.glob(*/)` to get subdirs was added in 3.11

    * bump zarr version to include resizing fix
    zarr-developers/zarr-python#1540

    * fix cli default

    * set log level with an environment variable

    * fix unset

    * catch non-existent page

    * implement fallback for incomplete channel names
    workaround for the Dragonfly microscope where the multi-camera setup only has one channel name written

    * add debug logs

    * handle virtual frames

    * try reading pages from TiffFile directly

    * filter error logs about ImageJ metadata being broken
    this is a known MM limitation when writing OME-TIFFs

    * fix regex

    * remove use of os.path in `convert.py`

    * better channel indexing in `_get_summary_metadata`

    * style

    * safer NoneType check

    * private default axis names for NDTiff

    * update documentation to reflect new entry point

    * add repr to MM FOV and dataset types

    * rename mm_meta and expose summary metadata

    * add MicroManagerFOVMapping.root

    * add MicroManagerFOVMapping.zyx_scale

    * add warning log for failed position grid

    * fix grid layout

    * suppress hypothesis flakiness

    * different health check suppression

    ---------

    Co-authored-by: Ivan Ivanov <[email protected]>

* black

* bugfix

---------

Co-authored-by: Ziwen Liu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ziw-liu ziw-liu ziw-liu approved these changes

Assignees
No one assigned
Labels
bug Something isn't working μManager Micro-Manager files and metadata
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ieivanov @ziw-liu