Embedding of non-image data?

[Voxelworld]

Hi guys

Is there a concept how to combine image data with non-image data in MONAI?

Let’s say I have an MRI scan and hi-res registered surface data, or an MRI scan and landmark positions. In this case it would make sense to preprocess and augment the unstructured data first in their domain and, if necessary, embed it in the discrete image grid only at the end. Is the transformation pipeline prepared for this?

In this scenarios, additional transformations would be necessary. For example, to convert landmark positions to gaussian heatmaps, or to generate an occupancy map of the mesh surface, so that a U-Net can directly be trained end-to-end on this data.

Even if I would implement the corresponding transformations, it is also not clear to me how I could synchronize transformations, like random augmentations, between images and landmarks. Or pass through the cropped region information (RandSpatialCropD/RandAffineD).

Thanks in advance.

[Nic-Ma]

Hi @ericspod and @wyli ,

I don't quite understand this feature request, why not put data into a dict and use the transform chain to handle it by keys?
Maybe it has some medical-specific challenge, could you please help share some comments?

Thanks in advance.

[ericspod]

For your application I think you would have to write additional transforms, specifically those working with dict data. The way I'd consider the problem is to define a transform which accepts a dictionary containing the image data in one set of keys and then other data like landmarks in other keys. Your transform would convert your landmarks to heatmaps or whatever other image representation you like, add those images as new keys or concatenate them with existing keys. The synchronization comes from the data for an image appearing in the same dict as the images themselves. Augmentation before this step would involve transforms which accept the same dict and modify the non-image data only. We don't have these transforms in MONAI yet but if you wanted to propose a feature request or their implementations we can look at integrating these in.

[Voxelworld]

Hi Eric

Thank you for your explanations. I'll admit the simplest solution is a feature request. :-)

Since I'm new to MONAI, I would still like to experiment with it and try to integrate my existing core algorithms into the framework.
Especially since I guess that it will take a while before MONAI can integrate landmark and surface support.
Unfortunately, I haven't fully understood your explanation yet, especially the point

"The way I'd consider the problem is to define a transform which accepts a dictionary containing the image data in one set of keys and then other data like landmarks in other keys."

It may be easier to clarify it with an example, e.g. a transformation pipeline with 2 types of data: volumes with registered landmarks (e.g. stored in DICOM coordinates).
I would expect the code to look something like this:

# Each dataset has a volume and associated landmark data:
datasets = [
{
   "volume"   : f"{id}.mha",
   "landmarks": f"{id}.csv"
} for id in some_ids
]

# All data should be randomly transformed and cropped:
transformations = Compose(
    LoadImageD(key="volume"),
    LoadLandmarksD(key="landmarks"),  #1. new transform
    ...
    RandAffineD(key="volume", rotate_range=np.pi, scale_range=0.2),    #2. currently only works on images...
    RandSpatialCropD(key="volume", roi_size=128, random_size=False),   #3. currently only works on images...
    ...
    RasterizeLandmarksD(key="landmarks", radius=2), #4. new transform
    ToTensor()
])

Obviously one would have to implement new transformations to 1. store the landmarks in the dict and 4. to rasterize the transformed landmarks finally to an image. As I understand, the latter could be done by using the physical image dimensions "_meta_dict" added by LoadImageDs.

Do I understand you correctly that you would suggest a new transformation class for each aug-transform (2. and 3.) with additional landmark support? Something like:

   MyCommonRandAffineD(image_key="volume", landmark_key="landmarks", ...)      # 2. works on all data types
   MyCommonRandSpatialCropD(image_key="volume", landmark_key="landmarks", ...) # 3. works on all data types

Do I have to rewrite this from scratch, or do you see a way to use the existing RandAffineD image transform?
Obviously, RandAffine code duplication would solve the problem, but this approach would also mean that the class has to be expanded each time an additional data type, e.g. surface data, should be supported. So the class would mix up many domains, which would be a bit ugly from "serparation of concerns" perspective. Otherwise a common transformation has to be used, so that a composition of the existing 'black box' class RandAffine seems to be dangerous regarding MONAI code changes.

Or is it possible to write a separate transformation for each data type and synchronize/copy only the part of RandomizableTransform.randomize? I haven't quite understood yet whether MONAIs concept of "determinism" covers that. But the approach would be a little more elegant:

transformations = Compose(
    ...
    RandAffineD(key="volume", ...).set_random_state(rseed)              #2a. still only images
    MyLandmarkRandAffineD(key="landmarks", ...).set_random_state(rseed) #2b. transforms landmarks with same 'randomize()'
    ...
])

Or should transformations better by synchronized over the common dict with well known-keys? Like:

    MyImageRandAffineD(key="volume",...)         # => update but also store d["volume_affine"] transformation
    MyLandmarkARandAffineD(key="landmarks",...)  # => read d["volume_affine"] and update d["landmarks"] accordingly

Unfortunately, this approach would then require a keyword convention in the dict, with hidden/magic keys, which would be prone to errors.

Best regards
Volker

[ericspod]

We do have a problem of generalizing how to do transforms on different types of data in one transform cleanly. We've been image focused and our transforms reflect this, what you want is to be able to transform image and landmarks in the same space, so applying an affine to the image has the same geometric effect on the landmarks. If we wanted to do this then yes we would need some new implementations which would take advantage of some parts of existing transforms at least. For affine transforms that would mean computing an affine matrix and applying it to the images and landmarks, this could reuse a lot of what's in RandAffineD but we still have the generalization problem again if some other new data type is then introduced.

With the dictionary transforms though they are applied only to those members of dictionaries whose keys are mentioned in the keys parameters, so to add noise to images only and leaving landmarks alone you can use existing transforms. You'll won't have to rewrite everything to do some operations. RasterizeLandmarksD will have to be implemented by you, but depending on what that would look like, and the network you're training, applying this earlier in your pipeline then applying transforms to the rasterized image might still work out.

There's still a lot to do here so it wouldn't be a bad idea to raise a feature request and see what can be done or who else might be interested in this as well.