Potential off-by-one when calculating resolution sizes #8

melissalinkert · 2024-09-10T00:11:57Z

Generate artificial pyramid using https://github.com/ome/bioformats/blob/develop/components/formats-gpl/utils/WritePyramid.java and:

$ java WritePyramid "0&sizeX=23073&sizeY=36336.fake" "1&sizeX=11537&sizeY=18168.fake" "2&sizeX=5769&sizeY=9084.fake" "3&sizeX=2885&sizeY=4542.fake" "4&sizeX=1443&sizeY=2271.fake" "5&sizeX=722&sizeY=1136.fake" "6&sizeX=361&sizeY=568.fake" "7&sizeX=181&sizeY=284.fake" ~/test-pyramid3.ome.tiff

then import to an OMERO server (qupath-testing ID 2128 is an example). Attempting to load the imported image in QuPath will throw FileNotFoundException when attempting to load level 2, which is resolution 5 according to Bio-Formats indexing. The level dimensions in QuPath are calculated to be 721x1137, which means that it's trying to read one more pixel in Y than actually exists.

Level dimensions are calculated according to the downsample factor reported by https://github.com/glencoesoftware/qupath-extension-omero-web/blob/main/src/main/java/qupath/lib/images/servers/omero/OmeroRequests.java#L83. The exact XY dimensions for each resolution level are not reported by that call. This is not simply a rounding issue, as the downsample factor times the largest resolution's SizeY is very slightly larger than 1137 (so Math.floor(...) wouldn't work). We'll need to get the exact XY dimensions of each resolution in some other way.

The text was updated successfully, but these errors were encountered:

melissalinkert · 2024-09-10T18:57:37Z

I feel like I must be missing something, but as far as I can tell there isn't way to get the XY size of each resolution without directly working with a PixelBuffer. I think that would mean some substantial changes to how this extension works, and would presumably require adding a dependency on https://github.com/ome/omero-romio (at minimum).

I think ideally, the request for resolution information:

qupath-extension-omero-web/src/main/java/qupath/lib/images/servers/omero/OmeroWebImageServer.java

Line 283 in d2cf2c3

JsonObject zoom = map.getAsJsonObject("zoomLevelScaling");

qupath-extension-omero-web/src/main/java/qupath/lib/images/servers/omero/OmeroWebImageServer.java

Line 209 in d2cf2c3

    
           JsonObject map = OmeroRequests.requestMetadata(scheme, host, port, Integer.parseInt(id));

qupath-extension-omero-web/src/main/java/qupath/lib/images/servers/omero/OmeroRequests.java

Line 83 in d2cf2c3

    
           public static JsonObject requestMetadata(String scheme, String host, int port, int id) throws IOException {

would give us not just the zoom level scale factor, but the actual XY sizes since those are available from PixelBuffer.getResolutionDescriptions(). If we can instead ask the image-region microservice for this information, then maybe a path forward would be to update this (I think?):

https://github.com/glencoesoftware/omero-ms-image-region/blob/master/src/main/java/com/glencoesoftware/omero/ms/image/region/ImageDataRequestHandler.java#L398

to include the resolution descriptions themselves in the returned JSON. Definitely would appreciate other opinions or better ideas though (@chris-allan / @sbesson).

https://forum.image.sc/t/qupath-omero-weird-pyramid-levels/65484 is some prior investigation on this topic, but to be clear I strongly disagree with the suggestion that this is something to fix in Bio-Formats.

sbesson · 2024-09-10T19:40:07Z

Thanks @melissalinkert, yes looking at this earlier, the OMERO.web imgData endpoint is definitely marshalling the resolutions descriptors returned by PixelBuffer.getResolutionDescriptions() into a one-dimensional array of downsampling factors for each resolution level. The relevant logic is in the OMERO.py gateway and notably ImageWrapper.getZoomLevelScaling() API where the dimension of each level alongside the X axis is divided by the sizeX of the highest resolution.

While having a simple scaling array makes sense from a viewer/zoom display perspective, I agree it is not sufficient to support scenarios like this one. Looking at the image.sc thread, it certainly like the QuPath / BioImageServer logic is internally working around this limitation as some formats are specially handled to account for the different of downsampling factors alongside X and Y - https://github.com/qupath/qupath/blob/17da581a49be1beb6af49a3cd15fa9c6f2527a95/qupath-extension-bioformats/src/main/java/qupath/lib/images/servers/bioformats/BioFormatsImageServer.java#L736-L745

I assume the problem is not limited to the micro-service though since the same resolution descriptors would be used when using the standard OMERO.web API? Do we know if this is failing in the same way for the synthetic image created above?

melissalinkert · 2024-09-11T16:38:46Z

I assume the problem is not limited to the micro-service though since the same resolution descriptors would be used when using the standard OMERO.web API?

Issue is definitely not limited to the microservices, and this extension currently requests zoom levels via /webgateway/imgData/. I was just thinking that updating the microservice and switching the extension to request resolution descriptions from the microservice might be a faster path to solving this, but maybe not?

I can view the imported image as expected without any errors in PathViewer and OMERO.web preview pane.

chris-allan · 2024-09-12T14:03:40Z

My sense here is that the reason this isn't particularly problematic for the viewers currently is that they make their requests via render_image_region which takes x, y, width, and height like tile. However, the behavior when the caller asks to read off the end is slightly different. So we've got a couple different ways of approaching a solution here:

Make tile more leanient like render_image_region (omero-ms-pixel-buffer)
Give the client more data in imgData so it doesn't read off the end in the first place (omero-web / omero-ms-image-region)

Combination of the two also an option since their not mutually exclusive.

melissalinkert · 2024-09-12T15:15:29Z

With this logging added:

$ git diff
diff --git a/src/main/java/qupath/lib/images/servers/omero/OmeroWebImageServer.java b/src/main/java/qupath/lib/images/servers/omero/OmeroWebImageServer.java
index 9477103..7f94fa7 100644
--- a/src/main/java/qupath/lib/images/servers/omero/OmeroWebImageServer.java
+++ b/src/main/java/qupath/lib/images/servers/omero/OmeroWebImageServer.java
@@ -207,6 +207,7 @@ public class OmeroWebImageServer extends AbstractTileableImageServer implements
                double magnification = Double.NaN;
                
                JsonObject map = OmeroRequests.requestMetadata(scheme, host, port, Integer.parseInt(id));
+    /* debug */ logger.warn("map = {}", map);
                JsonObject size = map.getAsJsonObject("size");
     JsonObject meta = map.getAsJsonObject("meta");
 
@@ -496,6 +497,7 @@ public class OmeroWebImageServer extends AbstractTileableImageServer implements
       URLConnection conn = url.openConnection();
       conn.setRequestProperty("Cookie", "sessionid=" + getWebclient().getSessionId());
       conn.connect();
+      /* debug */ logger.warn("{} request response = {}", urlFile, ((java.net.HttpURLConnection) conn).getResponseCode());
 
       BufferedImage img = ImageIO.read(conn.getInputStream());

I see this response from imgData:

map = {"id":2128,"meta":{"imageName":"test-pyramid3.ome.tiff","imageDescription":"","imageAuthor":"Import User","projectName":"Multiple","projectId":null,"projectDescription":"","datasetName":"Multiple","datasetId":null,"datasetDescription":"","wellSampleId":"","wellId":"","imageTimestamp":1725924331.0,"imageId":2128,"pixelsType":"uint8"},"perms":{"canAnnotate":true,"canEdit":true,"canDelete":true,"canLink":true},"tiles":true,"tile_size":{"width":256,"height":256},"levels":8,"zoomLevelScaling":{"0":1.0,"1":0.5000216703506263,"2":0.2500325055259394,"3":0.125037923113596,"4":0.06254063190742426,"5":0.0312919863043384,"6":0.0156459931521692,"7":0.007844666926710875},"interpolate":true,"size":{"width":23073,"height":36336,"z":1,"t":1,"c":1},"pixel_size":{"x":null,"y":null,"z":null},"init_zoom":0,"pixel_range":[0,255],"channels":[{"emissionWave":null,"label":"0","color":"808080","inverted":false,"reverseIntensity":false,"family":"linear","coefficient":1.0,"window":{"min":0.0,"max":255.0,"start":0.0,"end":255.0},"active":true}],"split_channel":{"g":{"width":23077,"height":36340,"border":2,"gridx":1,"gridy":1},"c":{"width":46152,"height":36340,"border":2,"gridx":2,"gridy":1}},"rdefs":{"model":"greyscale","projection":"normal","defaultZ":0,"defaultT":0,"invertAxis":false}}

Specifically, the zoom levels are:

"levels":8,"zoomLevelScaling":{"0":1.0,"1":0.5000216703506263,"2":0.2500325055259394,"3":0.125037923113596,"4":0.06254063190742426,"5":0.0312919863043384,"6":0.0156459931521692,"7":0.007844666926710875}

and 3 tiles with a response code other than 200:

/tile/2128/0/0/0?x=0&y=1024&w=256&h=113&format=tif&resolution=2 request response = 404
/tile/2128/0/0/0?x=256&y=1024&w=256&h=113&format=tif&resolution=2 request response = 404
/tile/2128/0/0/0?x=512&y=1024&w=209&h=113&format=tif&resolution=2 request response = 404

The actual exception that is thrown (server name replaced with $SERVER):

java.io.FileNotFoundException: $SERVER/tile/2128/0/0/0?x=0&y=1024&w=256&h=113&format=tif&resolution=2
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getChainedException(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
	at qupath.lib.images.servers.omero.OmeroWebImageServer.readTile(OmeroWebImageServer.java:502)
	at qupath.lib.images.servers.AbstractTileableImageServer.lambda$prerequestTiles$2(AbstractTileableImageServer.java:462)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at qupath.lib.images.servers.AbstractTileableImageServer.prerequestTiles(AbstractTileableImageServer.java:464)
	at qupath.lib.images.servers.AbstractTileableImageServer.readRegion(AbstractTileableImageServer.java:295)
	at qupath.lib.images.servers.AbstractTileableImageServer.readRegion(AbstractTileableImageServer.java:60)
	at qupath.lib.images.servers.AbstractImageServer.getDefaultThumbnail(AbstractImageServer.java:329)
	at qupath.lib.display.ImageDisplay.getImagesForHistogram(ImageDisplay.java:890)
	at qupath.lib.display.ImageDisplay.setImageData(ImageDisplay.java:238)
	at qupath.lib.gui.viewer.QuPathViewer.setImageData(QuPathViewer.java:1544)
	at qupath.lib.gui.QuPathGUI.openImage(QuPathGUI.java:1761)
	at qupath.lib.images.servers.omero.OmeroWebImageServerBrowserCommand.lambda$run$29(OmeroWebImageServerBrowserCommand.java:625)
	at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)
	at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:232)
	at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:189)
	at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
	at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49)
	at javafx.event.Event.fireEvent(Event.java:198)
	at javafx.scene.Scene$ClickGenerator.postProcess(Scene.java:3684)
	at javafx.scene.Scene$MouseHandler.process(Scene.java:3989)
	at javafx.scene.Scene.processMouseEvent(Scene.java:1890)
	at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2704)
	at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:411)
	at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:301)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:450)
	at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:424)
	at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:449)
	at com.sun.glass.ui.View.handleMouseEvent(View.java:551)
	at com.sun.glass.ui.View.notifyMouse(View.java:937)
	at com.sun.glass.ui.win.WinApplication._enterNestedEventLoopImpl(Native Method)
	at com.sun.glass.ui.win.WinApplication._enterNestedEventLoop(WinApplication.java:212)
	at com.sun.glass.ui.Application.enterNestedEventLoop(Application.java:515)
	at com.sun.glass.ui.EventLoop.enter(EventLoop.java:107)
	at com.sun.javafx.tk.quantum.QuantumToolkit.enterNestedEventLoop(QuantumToolkit.java:648)
	at javafx.stage.Stage.showAndWait(Stage.java:469)
	at qupath.lib.images.servers.omero.OmeroWebImageServerBrowserCommand.run(OmeroWebImageServerBrowserCommand.java:703)
	at qupath.lib.images.servers.omero.OmeroExtension.handleLogin(OmeroExtension.java:231)
	at qupath.lib.images.servers.omero.OmeroExtension.lambda$createServerListMenu$1(OmeroExtension.java:152)
	at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:86)
	at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:232)
	at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:189)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
	at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:49)
	at javafx.event.Event.fireEvent(Event.java:198)
	at javafx.scene.control.MenuItem.fire(MenuItem.java:459)
	at com.sun.javafx.scene.control.ContextMenuContent$MenuItemContainer.doSelect(ContextMenuContent.java:1385)
	at com.sun.javafx.scene.control.ContextMenuContent$MenuItemContainer.lambda$createChildren$12(ContextMenuContent.java:1338)
	at com.sun.javafx.event.CompositeEventHandler$NormalEventHandlerRecord.handleBubblingEvent(CompositeEventHandler.java:247)
	at com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:80)
	at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:232)
	at com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:189)
	at com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
	at com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
	at com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
	at com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)
	at javafx.event.Event.fireEvent(Event.java:198)
	at javafx.scene.Scene$MouseHandler.process(Scene.java:3980)
	at javafx.scene.Scene.processMouseEvent(Scene.java:1890)
	at javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2704)
	at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:411)
	at com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:301)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:450)
	at com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:424)
	at com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:449)
	at com.sun.glass.ui.View.handleMouseEvent(View.java:551)
	at com.sun.glass.ui.View.notifyMouse(View.java:937)
	at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
	at com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:185)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.FileNotFoundException: $SERVER/tile/2128/0/0/0?x=0&y=1024&w=256&h=113&format=tif&resolution=2
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
	at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
	at qupath.lib.images.servers.omero.OmeroWebImageServer.readTile(OmeroWebImageServer.java:500)
	... 88 more

chris-allan · 2024-09-12T15:37:02Z

I feel like we've got something else odd going on here because the math to produce that scaling factor does not make sense:

In [1]: 1136/36336
Out[1]: 0.03126376045794804

In [2]: 1137/36336
Out[2]: 0.03129128137384412

In [3]: 0.03126376045794804 * 36336
Out[3]: 1136.0

In [4]: 0.03129128137384412 * 36336
Out[4]: 1137.0

In [5]: 0.0312919863043384 * 36336
Out[5]: 1137.0256143544402

chris-allan · 2024-09-13T15:55:48Z

As @sbesson and @melissalinkert reminded me, the zoomLevelScaling metadata is calculated against X. When the downsampling results in non-even divisions between levels (ex 23073 / 2 = 11536.5) the zoomLevelScaling is obviously not a great way to reproduce the downsampling that happened in other dimensions. Unless they also did not divide evenly. For all off by one style cases I think the easiest way right now to avoid this is to work not with zoomLevelScaling directly but instead trying to reconstruct the original scaling factor. We can do this by taking the ratio between zoom levels and use that iteratively when attempting to reconstruct the factor used for dimensions other than X.

For example:

In [3]: size_x = 23073

In [4]: size_y = 36336

In [5]: zoom_level_scaling
Out[5]:
{'0': 1.0,
 '1': 0.5000216703506263,
 '2': 0.2500325055259394,
 '3': 0.125037923113596,
 '4': 0.06254063190742426,
 '5': 0.0312919863043384,
 '6': 0.0156459931521692,
 '7': 0.007844666926710875}

In [8]: resolutions
Out[8]:
['"0&sizeX=23073&sizeY=36336.fake"',
 '"1&sizeX=11537&sizeY=18168.fake"',
 '"2&sizeX=5769&sizeY=9084.fake"',
 '"3&sizeX=2885&sizeY=4542.fake"',
 '"4&sizeX=1443&sizeY=2271.fake"',
 '"5&sizeX=722&sizeY=1136.fake"',
 '"6&sizeX=361&sizeY=568.fake"',
 '"7&sizeX=181&sizeY=284.fake"']

In [38]: size_y * zoom_level_scaling['2']
Out[38]: 9085.181120790534

In [89]: factors = [1.0] + [round(zoom_level_scaling[str(a - 1)] / zoom_level_scaling[str(a)]) for a in range(1, 8)]

In [90]: factors
Out[90]: [1.0, 2, 2, 2, 2, 2, 2, 2]

In [91]: [math.ceil(size_y / math.prod(factors[:level])) for level in range(1, 9)]
Out[91]: [36336, 18168, 9084, 4542, 2271, 1136, 568, 284]

If the scaling factor is different between X and Y this won't improve anything and if it's fractional it won't work at all.

melissalinkert mentioned this issue Sep 13, 2024

Calculate resolution sizes based on an integer scale factor, not the zoom level #9

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Potential off-by-one when calculating resolution sizes #8

Potential off-by-one when calculating resolution sizes #8

melissalinkert commented Sep 10, 2024

melissalinkert commented Sep 10, 2024

sbesson commented Sep 10, 2024

melissalinkert commented Sep 11, 2024

chris-allan commented Sep 12, 2024

melissalinkert commented Sep 12, 2024

chris-allan commented Sep 12, 2024

chris-allan commented Sep 13, 2024 •

edited

Loading

Potential off-by-one when calculating resolution sizes #8

Potential off-by-one when calculating resolution sizes #8

Comments

melissalinkert commented Sep 10, 2024

melissalinkert commented Sep 10, 2024

sbesson commented Sep 10, 2024

melissalinkert commented Sep 11, 2024

chris-allan commented Sep 12, 2024

melissalinkert commented Sep 12, 2024

chris-allan commented Sep 12, 2024

chris-allan commented Sep 13, 2024 • edited Loading

chris-allan commented Sep 13, 2024 •

edited

Loading