Order of dimensions in torchio vs pytorch tensors

[Elmisiry99]

Hello,
I have been struggling with the issue of not knowing how does torchio saves the images it loads.
In the documentation of torchio, it is mentioned that when creating a ScalarImage(alone or in a subject), I have to pass either the path to the image or a 4D tensor. Normally, images are represented in pytorch as (B,C,D,H,W). On the other hand, torchio changes this order to (C,W,H,D), discarding the batch dimensions as it is not relevant in this case. My question is as follows:
Does torchio when input a tensor (C,D,H,W) rearranges it first to (C,W,H,D) or does it performs the transformation on the original tensor with the assumption that it is (C,W,H,D)?
I am concerned about this, as I am using torchio to define the dataset I use for the model I am working on and I am getting oscillating curves in the training loss. So I wanted to make sure that images I feed to the network are the right ones and I am not doing an indexing errors. I train the network using 2D slices sampled from the volumes I load using torchio. That is why I have to first apply the transforms on the whole 3D image, then slice this volume to access the desired 2D images.
Also I have another question regarding batches and torchio. Should I start a new discission for it or can I mention it here?
The following snippet contains an example of what I mean

def train_augmentation(self,
                       batch: torch.tensor):
         #batch is the batch passed by the dataloader and it is a tensor with the shape(B,C,D,H,W)
         image = batch['image'][tio.DATA][0]   # 'image' is name of the ScalarImage in the Subject. and I access only the first example of the batch for simplicity. image then has the shape (C,D,H,W)
        #My question is should I permute the tensor to (C,W,H,D) before any passing it to torchio transforms?              

         aug_1 = training_transform(image)
         aug_2 = training_transform(image)         
         print("aug_1 shape: ", aug_1.shape)
         print("aug_2 shape: ", aug_2.shape)

[fepegar]

Hi, @Elmisiry99. The concepts of width, height and depth are not very well defined for 3D images. The anatomical correspondence of each dimension depends on the images' orientation. I suggest you normalize their orientations using ToCanonical. That way, you'll be sure that dimensions will be (channels, left to right, posterior to anterior, inferior to superior).

---

Regarding your second question, you can just pass the transform to the dataset class, instead of performing it on the tensor extracted from the collated batch. Additionally, the typing hint is not correct, as that batch is expected to be a Dict[str, Dict[str, Any]].

[Elmisiry99]

Yes and I used them as a guide as well because I am using pytorchlightning and there is a tutorial for it. I define the dataset in the same way as in the TorchIO_MONAI_PyTorch-Lightning notebook. So when I pass the transforms, composed of preprocessing and augmentation, to the torchio.SubjectDataset as an argument, it will preprocess the image sampled form the dataset and augment it on the fly each time the dataloader samples an example from the dataset. Is that right?
If it is, then I don't need to subclass torch.utils.Dataset and can just work further with current implementation( as it follows the tutorial).

[fepegar]

So when I pass the transforms, composed of preprocessing and augmentation, to the torchio.SubjectDataset as an argument, it will preprocess the image sampled form the dataset and augment it on the fly each time the dataloader samples an example from the dataset. Is that right?

Exactly!

[Elmisiry99]

Thank you very much for the clarification, this explained a lot :).
I am working on a project to test a self-supervised based contrastive learning strategy. A huge part of the training relies on the idea of contrasting the representations of augmented images against each other, where I augment each image sampled twice before passing it to the network. This means that I just need to preprocess the images in the torchio.SubjectDataset and not include the augmentations. Augmentations can be applied later on each image of the batch separately to produce two perturbed views of each image. Pseudo code of the function where I augments the images of the batch:

def augment_images(batch: dict,transforms):
  new_images = []
  for each subject in the batch
     image = batch['images'] # Extract the images from the batch
     img_1 = transforms(image) # First augmented view
     img_2 = transforms(image) # Second augmented view
     new_images.append[img_1]
     new_images.append[img_2]
   return new_images

Please note that I don't need the labels for self-.supervised approach, that is why I don't use them.This is just to demonstrate the flow of the function, it is not real code. Do you think that applying the transforms like this to augment each image twice is the right way to do so or is there a better way?

[fepegar]

That approach seems inefficient, as the -expensive- augmentations are applied in the main process. Maybe a better way is to override __getitem__:

import copy
import torch
import torchio as tio


class MyDataset(tio.SubjectsDataset):
    def __getitem__(self, index: int):
        subject = self._subjects[index]
        subject = copy.deepcopy(subject)
        if self.load_getitem:
            subject.load()
        subject1 = self._transform(subject)
        subject2 = self._transform(subject)
        return subject1, subject2


colin = tio.datasets.Colin27()
subjects = [colin]
dataset = MyDataset(subjects, transform=tio.RandomAffine())
loader = torch.utils.data.DataLoader(
    dataset,
    num_workers=4,
    batch_size=1,
)
batch = tio.utils.get_first_item(loader)

[Elmisiry99]

I tried to override getitem() but I got this error:

TypeError: object of type 'type' has no len()

Code looks like this:

    training_dataset = tio.SubjectsDataset(training_2examples, transform=training_transforms) 

    class training_dataset(tio.SubjectsDataset):
        def __len__(self):
            return len(self._subjects)

        def __getitem__(self, index: int):
            subject = self._subjects[index]
            subject = copy.deepcopy(subject)
            if self.load_getitem:
                subject.load()
            
            subject1 = self._transform(subject)
            subject2 = self._transform(subject)

            return subject1, subject2

Could please tell me what is wrong here. Thanks in advance

[Elmisiry99]

Thank you so much for your help, you really clarified a lot of things that I didn't fully understand :).

[fepegar]

No worries, happy to help!

[fepegar]

Can you share the whole error message?

…

On Mon, 31 Jan 2022, 04:16 Elmisiry99, ***@***.***> wrote: I tried to override *getitem*() but I got this error: TypeError: object of type 'type' has no len() Code looks like this: training_dataset = tio.SubjectsDataset(training_2examples, transform=training_transforms) class training_dataset(tio.SubjectsDataset): def __len__(self): return len(self._subjects) def __getitem__(self, index: int): subject = self._subjects[index] subject = copy.deepcopy(subject) if self.load_getitem: subject.load() subject1 = self._transform(subject) subject2 = self._transform(subject) return subject1, subject2 Could please tell me what is wrong here. Thanks in advance — Reply to this email directly, view it on GitHub <#798 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADAZVVGFP3GZEGL6GZGXXMDUYYEIJANCNFSM5MP6ZOXQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Elmisiry99]

I don’t have the laptop with me right now, but I will post it as soon as I have it.
I noticed that I names the custom dataset trainingdataset but when I call it I still use torchio.SubjectDataset not trainingdatast. Do you think this maybe the reason?
And when I implement the len() function, should I return the length of self._subjects() or length of the augmented subjects. In the example above as I return two augmented versions so I mean with the latter choice that len() would return 2xlen(self._subjects).
Also I wanted to ask if I have to implement init() or not

[Elmisiry99]

The code is as follows:

class trainingDataSet(tio.SubjectsDataset):
  
    def __getitem__(self, index: int):
        subject = self._subjects[index]
        subject = copy.deepcopy(subject)
        if self.load_getitem:
            subject.load()
          
        subject1 = self._transforms(subject)
        subject2 = self._transforms(subject)
    
        return subject1, subject2
def PrepareDataset():
    training_trasforms= tio.Compose([
        tio.RandomNoise(p=0.5),
        tio.RandomFlip(),
        tio.RandomAffine()
            

    
    training_examples = tio.SubjectsDataset(subjects) # subjects is a list containing tio.Subjects
    training_dataset = trainingDataSet(subjects=training_examples, transform=training_transforms)  
    return training_dataset

train = PepareDataset(image_dir,label_dir)
loader = DataLoader(train, batch_size=1, num_workers=1)


for i, batch in enumerate(loader):
    print(i, batch['image'][tio.DATA].shape)

Now I get this error:

Traceback (most recent call last):
  File "/home/students/elmisiry/ba_elmisiry/Modularized Code/Data_preparation.py", line 152, in <module>
    for i, batch in enumerate(loader):
  File "/work/scratch/elmisiry/miniconda3/envs/pytorchnew/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/work/scratch/elmisiry/miniconda3/envs/pytorchnew/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/work/scratch/elmisiry/miniconda3/envs/pytorchnew/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/work/scratch/elmisiry/miniconda3/envs/pytorchnew/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/work/scratch/elmisiry/miniconda3/envs/pytorchnew/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/work/scratch/elmisiry/miniconda3/envs/pytorchnew/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/work/scratch/elmisiry/miniconda3/envs/pytorchnew/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/students/elmisiry/ba_elmisiry/Modularized Code/Data_preparation.py", line 28, in __getitem__
    subject1 = self._transforms(subject)
  File "/work/scratch/elmisiry/miniconda3/envs/pytorchnew/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 83, in __getattr__
    raise AttributeError
AttributeError

Please note that I made the changes I mentioned in the previous comment

[Elmisiry99]

Nevermind I made a mistake in calling the transforms in the getitem function