Multi-modality embedding

[karllandheer]

Hello, lets say I want to train an autoencoder for two different modalities. Maybe even one is 2D and the other is 3D. Is it possible to have some kind of joint embedding in an autoencoder via MONAI? Does anyone have any idea on how to go about this? Thanks very much in advance for any help!

[KumoLiu]

Hi @karllandheer, you may need to design a custom autoencoder architecture that can handle both 2D and 3D data. One approach could be to have separate encoders for each modality that map the input data to a shared embedding space. MONAI offers flexibility in building custom network architectures, making it possible to design such an autoencoder.
Thanks.

[karllandheer]

Hello KumoLiu, thank you very much for your kind reply. I agree that approach makes sense, however I'm unsure of how to actually create the joint embedding. From the autoencoder tutorial I see how an autoencoder is instantiated in MONAI:

https://github.com/Project-MONAI/tutorials/blob/main/modules/autoencoder_mednist.ipynb

It would be easy to make two separate ones, but how do I link their embeddings in MONAI? Could you provide any guidance? Thank you very much in advance,

Sincerely,
Karl