Error in training on timelapses

[KartikeyKansal1]

Hello, I'm trying to train on timelapses but the dataloader considers the time dimension as Z

I ran the following code and got the output below:

from aicsimageio import AICSImage

# Load the image using AICSImage
image_path = 'data/labelfree/training_data/Tension1.tif'
img = AICSImage(image_path)

# Print the shape of the image
print("Image shape:", img.shape)

# Print the number of dimensions
print("Image dimensions:", img.dims)

# Print the dimension order
print("Dimension order:", img.dims.order)

Image shape: (1, 1, 20, 512, 512)
Image dimensions: <Dimensions [T: 1, C: 1, Z: 20, Y: 512, X: 512]>
Dimension order: TCZYX

while the actual dimensions of the image are different:

I still wanna use 2D convolutions so I can't work with Z. Any suggestions?

I tried with conv3d and pulling out Z slices but that doesn't seem to work either.

[benjijamorris]

Hi @KartikeyKansal1! I haven't tried this exact thing, but my first thought is to incorporate the splitdim transformhttps://docs.monai.io/en/stable/transforms.html#splitdimd)

in something like:

transforms:
# loading image as Z(really T)YX
# split z dimension
_target_: monai.transforms.SplitDimd
keys: ${source_col}
dim: 1 # this is your Z/T dimension
list_output: True # this returns a list of dictionaries with the same keys as your original one

I think this will work, but your batches will be 20*batch_size and contain all the timepoints from a single image (so not i.i.d).

Another approach would be to use the aicsimageloader class. For this, you would create a csv with one row per timepoint like this:

path	T	C
/path1.tif	0	0}
/path1.tif	1	0
...	...	...
/path2.tif	0	0
/path2.tif	1	0

IO here might be slower, but you'll get better random sampling and have finer control over the batch size.

Let me know if either of these approaches works for you or if you want more details!

[KartikeyKansal1]

Hi @benjijamorris , thank you for the suggestions. I think I would like to go with the second approach here.

I'm assuming my data config file should look something like this

train:
    _target_: monai.transforms.Compose
    transforms:
      - _target_: monai.transforms.LoadImaged
        keys: ${target_col}
        reader:
          - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
            dimension_order_out: YX
            C: 0
            Z: 0

I'm confused on how to create this csv, how will I pair the source and target column.

[benjijamorris]

Good question! This approach would replace the LoadImaged Transform, so

train:
    _target_: monai.transforms.Compose
    transforms:
        - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
          path_key: seg_path # key in your csv with the path to your segmentation
          out_key: seg # name to load your segmentation into

        - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
          path_key: raw_path # key in your csv with the path to your paired raw image (i.e. in the same row in your csv)
          out_key: raw # name to load your raw image into

This will give you a dictionary like

{'raw': np.array(), 'seg': np.array(), ... }

your csv would look like this:

seg_path	raw_path	T	C	dimension_order_out
/seg_path1.tif	/raw_path1.tif	0	0	ZYX
/seg_path1.tif	/raw_path1.tif	1	0	ZYX

[KartikeyKansal1]

Gotcha,

Will rest of the transforms can still be applied to ${data.columns}? and I can define columns as

columns:
  - ${seg_col}
  - ${raw_col}
  - ${T_col}
  - ${C_col}
  - ${dim_col}

And spatial_dims=3 since ZYX?

[benjijamorris]

I would change ${data.columns} to a yaml list like (otherwise the transforms will try to augment the column values that aren't images as well)

- ${seg_col}
- ${raw_col}

spatial dims should be 2 since your model inputs will be CYX

[KartikeyKansal1]

Getting this error:

Error executing job with overrides: ['experiment=im2im/labelfree.yaml', 'trainer=cpu', 'spatial_dims=2']
Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/spatial/dictionary.py", line 1991, in __call__
    d[key] = self.zoomer(
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/spatial/array.py", line 1092, in __call__
    _zoom = ensure_tuple_rep(self.zoom, img.ndim - 1)  # match the spatial image dim
AttributeError: 'str' object has no attribute 'ndim'

I'm also not sure why the dimension_order_out is ZYX in the table.

[benjijamorris]

Looks like the zoom transform is operating on a path instead of the loaded image. I'd make sure that whatever you choose as the out_key is the key that's being augmented in later transforms.

Good point, that should be CYX

[KartikeyKansal1]

I removed the zoom transform

Error I'm getting now:

File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/utils/type_conversion.py", line 332, in convert_data_type
    raise ValueError(f"Unsupported output type: {output_type}")
ValueError: Unsupported output type: <class 'str'>

My data config file:

_target_: cyto_dl.datamodules.dataframe.DataframeDatamodule

path: 
cache_dir:

num_workers: 0
batch_size: 1
pin_memory: True
split_column: 
columns:
  - ${raw_col}
  - ${signal_col}

transforms:
  train:
    _target_: monai.transforms.Compose
    transforms:
      - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
        path_key: raw_path # key in your csv with the path to your paired raw image (i.e. in the same row in your csv)
        out_key: raw # name to load your raw image into

      - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
        path_key: signal_path # key in your csv with the path to your paired signal image (i.e. in the same row in your csv)
        out_key: signal # name to load your signal image into

      - _target_: monai.transforms.ToTensord
        keys: 
          - ${raw_col}
          - ${signal_col}
      - _target_: monai.transforms.NormalizeIntensityd
        keys: 
          - ${raw_col}
          - ${signal_col}
        channel_wise: True
      - _target_: cyto_dl.image.transforms.RandomMultiScaleCropd
        keys: 
          - ${raw_col}
          - ${signal_col}
        patch_shape: ${data._aux.patch_shape}
        patch_per_image: 1
        scales_dict: ${kv_to_dict:${data._aux._scales_dict}}

  test:
    _target_: monai.transforms.Compose
    transforms:
      - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
        path_key: raw_path # key in your csv with the path to your paired raw image (i.e. in the same row in your csv)
        out_key: raw # name to load your raw image into

      - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
        path_key: signal_path # key in your csv with the path to your paired signal image (i.e. in the same row in your csv)
        out_key: signal # name to load your signal image into

      - _target_: monai.transforms.ToTensord
        keys: 
          - ${raw_col}
          - ${signal_col}
      - _target_: monai.transforms.NormalizeIntensityd
        keys: 
          - ${raw_col}
          - ${signal_col}

  predict:
    _target_: monai.transforms.Compose
    transforms:
      - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
        path_key: raw_path # key in your csv with the path to your paired raw image (i.e. in the same row in your csv)
        out_key: raw # name to load your raw image into

      - _target_: monai.transforms.ToTensord
        keys: ${raw_col}
      - _target_: monai.transforms.NormalizeIntensityd
        keys: ${raw_col}
        channel_wise: True

  valid:
    _target_: monai.transforms.Compose
    transforms:
      - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
        path_key: raw_path # key in your csv with the path to your paired raw image (i.e. in the same row in your csv)
        out_key: raw # name to load your raw image into

      - _target_: cyto_dl.image.io.aicsimage_loader.AICSImageLoaderd
        path_key: signal_path # key in your csv with the path to your paired signal image (i.e. in the same row in your csv)
        out_key: signal # name to load your signal image into

      - _target_: monai.transforms.ToTensord
        keys: 
          - ${raw_col}
          - ${signal_col}
      - _target_: monai.transforms.NormalizeIntensityd
        keys: 
          - ${raw_col}
          - ${signal_col}
        channel_wise: True
      - _target_: cyto_dl.image.transforms.RandomMultiScaleCropd
        keys: 
          - ${raw_col}
          - ${signal_col}
        patch_shape: ${data._aux.patch_shape}
        patch_per_image: 1
        scales_dict: ${kv_to_dict:${data._aux._scales_dict}}

_aux:
  _scales_dict:
    - - ${raw_col}
      - [1]
    - - ${signal_col}
      - [1]

[benjijamorris]

Hmm, I bet it's in ToTensord, maybe specifying dtype: float16 will help (if you're using mixed precision, otherwise float32)

[KartikeyKansal1]

Still getting the error:

Traceback (most recent call last):
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/intensity/dictionary.py", line 813, in __call__
    d[key] = self.normalizer(d[key])
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/intensity/array.py", line 910, in __call__
    img[i] = self._normalize(  # type: ignore
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/transforms/intensity/array.py", line 864, in _normalize
    img, *_ = convert_data_type(img, dtype=torch.float32)
  File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/monai/utils/type_conversion.py", line 332, in convert_data_type
    raise ValueError(f"Unsupported output type: {output_type}")
ValueError: Unsupported output type: <class 'str'>

The above exception was the direct cause of the following exception:

[benjijamorris]

ah sorry totally missed this.
changing out_key: raw to out_key: ${raw_col} and out_key: signal to out_key: ${signal_col} should do it.

Right now, unless your raw_col is "raw" and signal_col is "signal", when you reference ${signal_col} it will be the string in that column of your csv (not the image you loaded in the image loading step). Is this making sense?

I want to double check with you too that path_key: raw_path is correct, not path_key: ${raw_col}

[KartikeyKansal1]

I made the changes. And correct, path_key: raw_path. Thanks for updating and checking in!

Error:

File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [1, 1, 0, 0, 512, 512]

[benjijamorris]

Okay looks like some extra channels are getting introduced there...
You could try changing 'CYX' to 'YX' in your csv. What is your data.aux.patch_shape? Is it 2d or 3d?

[KartikeyKansal1]

I changed it YX but still getting the same error. 2D

[benjijamorris]

I'd try clearing your cache_dir (if you're using it). Is your patch_shape [512,512]? I think the 0-dimensions are coming from the cyto_dl.image.transforms.RandomMultiScaleCropd, if you add the monai.transforms.DataStatsD to your list of transformer before cropping, are the tensors the shape you expect? (should be [1, 512, 512] I think)

[KartikeyKansal1]

Thank you for the suggestion.

[2024-07-16 19:55:49,661][DataStats][INFO] - Data statistics:
Type: <class 'monai.data.meta_tensor.MetaTensor'> torch.float16
Shape: torch.Size([1, 1, 20, 512, 512])
Value range: (22416.0, 46592.0)
[2024-07-16 19:55:49,667][DataStats][INFO] - Data statistics:
Type: <class 'monai.data.meta_tensor.MetaTensor'> torch.float16
Shape: torch.Size([1, 1, 20, 512, 512])
Value range: (1578.0, 16800.0)
[2024-07-16 19:55:49,673][DataStats][INFO] - Data statistics:
Type: <class 'monai.data.meta_tensor.MetaTensor'> torch.float16
Shape: torch.Size([1, 1, 20, 512, 512])
Value range: (22416.0, 46592.0)
[2024-07-16 19:55:49,678][DataStats][INFO] - Data statistics:
Type: <class 'monai.data.meta_tensor.MetaTensor'> torch.float16
Shape: torch.Size([1, 1, 20, 512, 512])
Value range: (1578.0, 16800.0)

The cache is clear.
I'm still getting the same error:

File "/Users/kartikeykansal/miniconda3/envs/cytodl_env_240711_v2/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
  return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [1, 1, 0, 0, 512, 512]

My patch shape is [256,256]

[benjijamorris]

Okay, it looks like the issue is in the aicsimageloaderd transform. I think you want the shape to be [1, 512, 512] CYX instead of [1, 1, 20, 512, 512] (I think this is [something something TYX]). I'm not sure what exactly your metadata or csv with image loading parameters look like, but I think messing with the csv so you're just loading CYX from each row in your csv will solve the problem.