Reading in patches from file directly

[Jarartur]

Hi,
I am dealing with histopathology 2D images, which I found torchio to be perfect! I wanted to optimize inference memory-wise with GridAggregator.
The way I understand it, torchio needs to first load the whole image into RAM, and only then it can extract patches and push them to the GPU VRAM. Is it true? I am talking from the perspective of libraries like OpenSlide and pyvips which can read in a patch at a time.
Would it be possible to write a custom reader that will read in only the specified region of the image? I've thought about returning a custom class from the Subject and calling it with tio.LOCATION, so something like:

subject = ... # a class with some width and height attributes

grid_sampler = tio.inference.GridSampler(
    subject,
    self.patch_size,
    patch_overlap,
)

patch_loader = tc.utils.data.DataLoader(grid_sampler, batch_size=batch_size)
aggregator = tio.inference.GridAggregator(grid_sampler)

for patches_batch in patch_loader:
    input_tensor = patches_batch["src"].load_patch([tio.LOCATION])

But I don't really know if it's going to fit with torchio model.

[romainVala]

Hello
It looks to me like an important improvement to do. It will be nice if you have time to implement it

yes the way sampler are written assume the data is read within the Subject (data key). So GridSampler for instance will set a given location and will extract directly from the tensor data stored in torchio Subject.

If I understand correctly what you propose, this imply to change not only the Subject class but also all Sampler (so that instead of getting a subpart of subject tensor data it will call the subject.load_path at a given location
may be it will not be to much change (mainly in extract_path function define in sampler.py)

but I am not sure ... may be @fepegar have a better overview of this issue ?

An other option is may be to define a new DataLoader for you histo data. Instead to have on subject per file you can implement a method, that give a subject (containing a patch of you all image at a random / given location)
This will then mean you have to re-implement all the Sampler logic in your DataLoader. So it is may be not an ideal solution ...
the advantage though, is that you can then have two patch size level. A first one bigger (for instance 512*512) that you use to construct the subject data. Then you can add torchio transform to it. and then you could use the torchio patch (with smaller size) via sampler or via torchio.queue.
The advantage of this second option is for the transform (especially spatial transform). In the current model transform are perform at the subject level and not at the patch one, so to use torchio transform it may be better to choose this option. For rotation or translation it is better to apply it to a bigger field of view and then take one or several patches (so that you can get rid off side effects)

I hope I am clear enough, although I am not so sure, but it is worth to specify a bit what the objectives are. An important question is when to apply the torchio transforms ?

[Jarartur]

Hi, and thanks for the comprehensive answer, it is indeed clear.
This looks like an interesting thing to work on, but unfortunately, I am currently racing to finish my master's thesis on time. I will probably create something simpler and less generalizable for the moment.
For WSIs (histopathological images) I do not see an easy way to create a PR without creating additional dependencies (like pyvips) but maybe it is doable with SimpleITK.

I might just return to this problem after my master's. Perhaps with the insight I will have gained by then, I will work on proposing an integration to torchio as I like the library immensely :).

[romainVala]

I think it is an important improvement for the histopathology community
adding a dependence to read standard histo file format seems to me ok (similar to nibabel for MRI nifti files)

I understand making a proper PR will be more work, but do not hesitate to copy past your quick solution, it may help future dev on this topic

[fepegar]

The Crop transform reads the data attribute for slicing. Maybe something like this (but adapted to WSIs) would help:

import nibabel as nib
import torchio as tio

img_path = tio.datasets.Colin27().t1.path

class LazyImage:
    def __init__(self, path):
        self._img = nib.load(path)
        self.data = self._img.dataobj

lazy_image = LazyImage(img_path)
lazy_image.data[10:12, 20:23]

array([[[353901.94043323, 386856.37203756, 379041.10352266, ...,
          14849.03239323,  12374.19390786,  12634.70384569],
        [321208.02172872, 356897.8150852 , 420983.11169106, ...,
          34647.7447192 ,  26441.69796861,  17975.14572317],
        [394020.38199884, 423588.19329739, 414079.59093368, ...,
          54316.19911427,  44156.33523495,  45068.11853635]],

       [[356354.79947558, 375317.05337393, 383579.17947158, ...,
          11512.79256673,  10564.68046421,  10835.56995371],
        [361095.35702619, 393737.52385029, 407011.09920955, ...,
          26411.70190102,  26411.70190102,  12596.35052475],
        [368003.04826477, 373014.49752638, 416356.76845527, ...,
          62710.86239374,  45915.72737341,  39685.27429829]]])

I suppose you could add a subclass of Image that never loads the image and returns slices using something like the above. MONAI has good wrappers for WSIs, so those could probably be leveraged together with TorchIO.

[fepegar]

Others that might be interested are @MirenLurBarquin, @jxchen01, @jmlipman

[Jarartur]

Oh, that's a good idea. I will start with that.