20230922-minutes.md

Imaging Working Group Minutes

Thursday, 2023/09/22

10:00 am ET

Attendees:

Terrell Russell, Kory Draughn, Tatiana Woller (KU Leuven), Tony Edgin (CyVerse / UArizona), Alice Stuart-Lee (SURF), Ken Ho (Crick, UK), Marjolijn Mertz (NKI - Netherlands Cancer Institute), Josh Moore (GerBI / OMERO), Ingrid Barcena (KU Leuven), Nirav Merchant (CyVerse / UArizona)

Minutes

• BISQUE

  • https://bioimage.ucsb.edu/bisque
  • Fetching of remote files
  • Lots of caching
  • Bisque-specific metadata is kept in bisque
  • UUID-tied to bisque is kept in iRODS AVU
  • Uploads to bisque go into the users' irods bisque_data collection
  • One bisque object is one physical file / iRODS object

• OMERO

  • One omero object can point to multiple files
  • would / could also use an internal format (bioformat-based) like bisque
  • Largely has been assuming no duplication of files, and relatively small datasets
    • Fits on a single disk

• STUFF WE TRIED/DID

  • Planning to parse some XML of OMERO objects
    • Parse to interesting AVU strings
    • Send to iRODS
  • conda install -c ome bftools
  • showinf -nopix -omexml-only a.fake

• USE CASES

  • A) Analysis in OMERO - read-only(?) from a data store (iRODS/S3/others)
    • Teach iRODS how to register/send info into OMERO brain
    • Synchronization of any work back into iRODS brain
  • B) Upload to OMERO - would write to iRODS (no multiple copies of big files, and abstracts the storage to different backends / locations)
  • C) Annotation in OMERO - perhaps save/write metadata into iRODS (for visibility)
  • D) Annotation in iRODS - metadata visible in OMERO?
  • E) Teach iRODS how to push things to OMERO
  • Bisque does A and C? Just A?
  • Everyone here wants A through D, equally.
    • Reading and writing are both important

• Possible serialization of OMERO object information

  • Just write it down as a file - usually for an individual user
  • No new code, just 'export/serialize' into iRODS
  • Perhaps an AVU or two for identification / lookup
  • Philosophical Breakthrough may be thinking/defining this as a 'snapshot'
  • AA) 'OMERO-transfer'
    • Creates a tree of files
      • Metadata files at the top
      • Tree structure matches the originals in OMERO
    • Pack, send everything, perhaps tarred/zipped
    • Disaster recovery
    • Migration to a different OMERO
  • BB) 'unpack' from iRODS back to OMERO

    • Reconstitute a serialization

      $ unzip -l valid_single_dataset.zip
      Archive:  valid_single_dataset.zip
        Length      Date    Time    Name
      ---------  ---------- -----   ----
              0  02-17-2022 12:22   root_0/
           2100  02-17-2022 12:22   transfer.xml
              0  02-17-2022 12:22   root_0/2022-01/
              0  02-17-2022 12:22   root_0/2022-01/14/
              0  02-17-2022 12:22   root_0/2022-01/14/18-30-48.537/
          78672  02-17-2022 12:22   root_0/2022-01/14/18-30-48.537/cycle4.ome_cells.png
      

• https://github.com/ome/omero-cli-transfer/blob/main/test/data/valid_single_dataset.zip

• Lots of links

  • https://gitlab.in2p3.fr/fbi-data/fbi-omero
  • https://github.com/irods-contrib/irods_working_group_imaging/blob/main/sandbox/docker-compose.yml
  • https://github.com/ome/omero-cli-transfer/blob/main/src/generate_xml.py
  • https://github.com/cyverse/irodsfs
  • Proxying… (like sudo)
  • https://github.com/irods/python-irodsclient#simple-puts-and-gets (parallel)
  • https://github.com/irods/python-irodsclient#reading-and-writing-files (stream)
  • https://superfastpython.com/multithreaded-zip-files/
  • https://www.libarchive.org/

• Script in OMERO

  • OMERO possibly stores user’s name in their account (but some mapping)
  • User has access to script and run pip install omero-cli-transfer irodsclient
    • moore->omero
    • Jackson lab - Dr. Erick Martins Ratamero -> omero
      • https://erickmartins.github.io/
  • User clicks on something/button/export/sendtoirods
    • Maybe no form to fill in (username/target/etc)
  • This runs omero transfer pack –server-side some.zip
  • Script copies some.zip to target location
  • Then iRODS perhaps runs any policy to expand/permission/extractAVUs

• Future work to have it NOT be a direct injection, but rather go through an agent?

  • Agent would have admin permissions in 'both' directions
  • Easier to keep all omero/irods related code in lock-step

• Separate effort

  • F) user migrating to different OMERO
  • 'PULL' back zip files from iRODS
  • Hydrate into the OMERO side of things

• Alternate alternate possible future

  • No NFSRODS anymore
  • ONLY transfer in and transfer out w/ metadata capabilities
  • Just let OMERO do OMERO things with local install

• Next Meeting:

  • Oct 19