load data with a 'CustomLoadImaged' but error occurs when using 'num_works'

[ChenchenHu007]

hi, everyone. I am a beginner of Monai.
I want to load data using SimpleITK, but the Monai does not support it.

So a CustomLoadImaged is defined:

class CustomLoadImaged:
    def __call__(self, patient_dir, slice_index: Optional[int] = None):
        if not slice_index:
            list_files = os.listdir(patient_dir)
            slice_index = int(random.sample(list_files, 1)[0].split('_')[-1].split('.')[0])
        d = dict()
        d['case_id'] = int(patient_dir.split('_')[-1])
        d['slice_index'] = slice_index
        list_structures = ['CT', 'mask']

        for structure_name in list_structures:
            structure_file = patient_dir + '/' + structure_name + '_' + str(slice_index) + '.nii.gz'

            if structure_name == 'CT':
                dtype = sitk.sitkInt16
            else:
                dtype = sitk.sitkUInt8

            if os.path.exists(structure_file):
                d[structure_name] = sitk.ReadImage(structure_file, dtype)
                d[structure_name] = sitk.GetArrayFromImage(d[structure_name])
            else:
                d[structure_name] = np.zeros((1, 128, 128), np.uint8)

        return d

the dataset, loader and its transforms are defined as follow:

train_transforms = monai.transforms.Compose([
    monai.transforms.adaptor(CustomLoadImaged(), outputs=None,
                             inputs={'patient_dir': 'patient_dir'}),
])

train_case = [{'patient_dir': patient} for patient in patients[:200]]

train_ds = monai.data.Dataset(train_case, transform=train_transforms)
train_loader = torch.utils.data.DataLoader(train_ds,
                                     batch_size=40,
                                     shuffle=True,
                                     )

when I iterated the loader, I printed how many iterations in it like this. it works.

 for index, batch_data in enumerate(train_loader):
     print(index)

However, when I added num_workers=2 in the train_loader, a problem occurred.
---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-24-d524bf703244> in <module>
----> 1 for index, batch_data in enumerate(train_loader):
      2     print(index)

~/opt/miniconda3/envs/monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py in __iter__(self)
    357             return self._iterator
    358         else:
--> 359             return self._get_iterator()
    360 
    361     @property

~/opt/miniconda3/envs/monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py in _get_iterator(self)
    303         else:
    304             self.check_worker_number_rationality()
--> 305             return _MultiProcessingDataLoaderIter(self)
    306 
    307     @property

~/opt/miniconda3/envs/monai/lib/python3.9/site-packages/torch/utils/data/dataloader.py in __init__(self, loader)
    916             #     before it starts, and __del__ tries to join but will get:
    917             #     AssertionError: can only join a started process.
--> 918             w.start()
    919             self._index_queues.append(index_queue)
    920             self._workers.append(w)

~/opt/miniconda3/envs/monai/lib/python3.9/multiprocessing/process.py in start(self)
    119                'daemonic processes are not allowed to have children'
    120         _cleanup()
--> 121         self._popen = self._Popen(self)
    122         self._sentinel = self._popen.sentinel
    123         # Avoid a refcycle if the target function holds an indirect

~/opt/miniconda3/envs/monai/lib/python3.9/multiprocessing/context.py in _Popen(process_obj)
    222     @staticmethod
    223     def _Popen(process_obj):
--> 224         return _default_context.get_context().Process._Popen(process_obj)
    225 
    226 class DefaultContext(BaseContext):

~/opt/miniconda3/envs/monai/lib/python3.9/multiprocessing/context.py in _Popen(process_obj)
    282         def _Popen(process_obj):
    283             from .popen_spawn_posix import Popen
--> 284             return Popen(process_obj)
    285 
    286     class ForkServerProcess(process.BaseProcess):

~/opt/miniconda3/envs/monai/lib/python3.9/multiprocessing/popen_spawn_posix.py in __init__(self, process_obj)
     30     def __init__(self, process_obj):
     31         self._fds = []
---> 32         super().__init__(process_obj)
     33 
     34     def duplicate_for_child(self, fd):

~/opt/miniconda3/envs/monai/lib/python3.9/multiprocessing/popen_fork.py in __init__(self, process_obj)
     17         self.returncode = None
     18         self.finalizer = None
---> 19         self._launch(process_obj)
     20 
     21     def duplicate_for_child(self, fd):

~/opt/miniconda3/envs/monai/lib/python3.9/multiprocessing/popen_spawn_posix.py in _launch(self, process_obj)
     45         try:
     46             reduction.dump(prep_data, fp)
---> 47             reduction.dump(process_obj, fp)
     48         finally:
     49             set_spawning_popen(None)

~/opt/miniconda3/envs/monai/lib/python3.9/multiprocessing/reduction.py in dump(obj, file, protocol)
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60     ForkingPickler(file, protocol).dump(obj)
     61 
     62 #

AttributeError: Can't pickle local object 'adaptor.<locals>._inner'

I can't solve this problem. Would anyone can help me ?
Here is my monai config:

MONAI version: 0.5.3
Numpy version: 1.20.2
Pytorch version: 1.9.0
MONAI flags: HAS_EXT = False, USE_COMPILED = False
MONAI rev id: d78c669c67e38ddfbe572f6a0438e9df0b8c65d7

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: 3.2.1
scikit-image version: NOT INSTALLED or UNKNOWN VERSION.
Pillow version: 8.2.0
Tensorboard version: 2.5.0
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.10.0
ITK version: 5.2.0
tqdm version: 4.59.0
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: NOT INSTALLED or UNKNOWN VERSION.

[Nic-Ma]

Hi @ChenchenHu007 ,

Thanks very much for your bug report.
I think our transform adapter utility may not support multi-processing, let me try to make a quick PR for it.

Thanks.

[Nic-Ma]

Hi @ChenchenHu007 ,

May I know your OS ?
I tried to run a DataLoader with adaptor transform with 2 workers locally, didn't see any issue in ubuntu.
We don't suppose it to work in multi-processing of windows or mac.

Thanks.

[ChenchenHu007]

hi ma, thanks for your reply . The error occurred when I ran those code locally on mac. However when I run it on colab, it works fine.

[Nic-Ma]

Cool, sounds like it's as expected.

Thanks.