Cannot pass nnUNetv2_plan_and_preprocess -d 002 --verify_dataset_integrity #1409

yejin2008 · 2023-04-19T22:14:36Z

yejin2008
Apr 19, 2023

I download heart dataset from: https://drive.google.com/drive/folders/1HqEgzS8BV2c7xYNrZdEAnrHk7osJJ--2
perform the data conversion: nnUNetv2_convert_old_nnUNet_dataset /nnUNet/dataset/nnUNet_raw/Task02_Heart Dataset002_Heart
run:
nnUNetv2_plan_and_preprocess -d 002 --verify_dataset_integrity
Got error:
Fingerprint extraction...
Dataset002_Heart
dataset directory:/nnUNet/dataset/nnUNet_raw/Dataset002_Heart
Traceback (most recent call last):
File "/nnUNet/nnUNetv2_plan_and_preprocess", line 33, in
sys.exit(load_entry_point('nnunetv2', 'console_scripts', 'nnUNetv2_plan_and_preprocess')())
File "/nnUNet/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_entrypoints.py", line 173, in plan_and_preprocess_entry
extract_fingerprints(args.d, args.fpe, args.npfp, args.verify_dataset_integrity, args.clean, args.verbose)
File "/nnUNet/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 46, in extract_fingerprints
extract_fingerprint_dataset(d, fingerprint_extractor_class, num_processes, check_dataset_integrity, clean,
File "/nnUNet/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 29, in extract_fingerprint_dataset
verify_dataset_integrity(join(nnUNet_raw, dataset_name), num_processes)
File "/nnUNet/nnUNet/nnunetv2/experiment_planning/verify_dataset_integrity.py", line 158, in verify_dataset_integrity
assert len(training_identifiers) == expected_num_training, 'Did not find the expected number of training cases '
AssertionError: Did not find the expected number of training cases (20). Found 1 instead.
Examples: ['l']

NajaJean · 2023-04-24T13:09:11Z

NajaJean
Apr 24, 2023

I had a similar problem, but with the 001BrainTumor dataset. I got around the issue by changing the the input when using the function determine_reader_writer_from_dataset_json() as '0000' were added manually to the name there (which I believe is the root of the problem). I now have the problem when running:
C:\WINDOWS\system32> nnUNetv2_plan_and_preprocess -d 1 --verify_dataset_integrity

Fingerprint extraction...
Dataset001_BrainTumour
Using <class 'nnunetv2.imageio.simpleitk_reader_writer.SimpleITKIO'> as reader/writer
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Users\-\Anaconda3\envs\unet\Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\c-\Anaconda3\envs\unet\Lib\multiprocessing\pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\-\Anaconda3\envs\unet\Lib\site-packages\nnunetv2\experiment_planning\verify_dataset_integrity.py", line 56, in check_cases
    images, properties_image = rw.read_images(files_image)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\-\Anaconda3\envs\unet\Lib\site-packages\nnunetv2\imageio\simpleitk_reader_writer.py", line 100, in read_images
    stacked_images = np.vstack(images)
                     ^^^^^^^^^^^^^^^^^
  File "<__array_function__ internals>", line 200, in vstack
  File "C:\Users\cmg818\Anaconda3\envs\unet\Lib\site-packages\numpy-1.24.2-py3.11-win-amd64.egg\numpy\core\shape_base.py", line 296, in vstack
    return _nx.concatenate(arrs, 0, dtype=dtype, casting=casting)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<__array_function__ internals>", line 200, in concatenate
ValueError: need at least one array to concatenate
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\-\Anaconda3\envs\unet\Scripts\nnUNetv2_plan_and_preprocess.exe\__main__.py", line 7, in <module>
  File "C:\Users\-\Anaconda3\envs\unet\Lib\site-packages\nnunetv2\experiment_planning\plan_and_preprocess_entrypoints.py", line 173, in plan_and_preprocess_entry
    extract_fingerprints(args.d, args.fpe, args.npfp, args.verify_dataset_integrity, args.clean, args.verbose)
  File "C:\Users\-\Anaconda3\envs\unet\Lib\site-packages\nnunetv2\experiment_planning\plan_and_preprocess_api.py", line 46, in extract_fingerprints
    extract_fingerprint_dataset(d, fingerprint_extractor_class, num_processes, check_dataset_integrity, clean,
  File "C:\Users\-\Anaconda3\envs\unet\Lib\site-packages\nnunetv2\experiment_planning\plan_and_preprocess_api.py", line 29, in extract_fingerprint_dataset
    verify_dataset_integrity(join(nnUNet_raw, dataset_name), num_processes)
  File "C:\Users\-\Anaconda3\envs\unet\Lib\site-packages\nnunetv2\experiment_planning\verify_dataset_integrity.py", line 196, in verify_dataset_integrity
    result = p.starmap(
             ^^^^^^^^^^
  File "C:\Users\-\Anaconda3\envs\unet\Lib\multiprocessing\pool.py", line 375, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\-\Anaconda3\envs\unet\Lib\multiprocessing\pool.py", line 774, in get
    raise self._value
ValueError: need at least one array to concatenate

0 replies

FabianIsensee · 2023-05-02T13:32:03Z

FabianIsensee
May 2, 2023
Maintainer

@yejin2008 you need to use nnUNetv2_convert_MSD_dataset, not nnUNetv2_convert_old_nnUNet_dataset. The datase you downloaded is in the MSD format. This could also be the cause of your problem @NajaJean

0 replies

yejin2008 · 2023-05-02T15:13:34Z

yejin2008
May 2, 2023
Author

I used: nnUNetv2_convert_MSD_dataset -i /nnUNet/dataset/nnUNet_raw/Task02_Heart. to convert Then nnUNetv2_plan_and_preprocess -d 002 --verify_dataset_integrity, still got error: Fingerprint extraction... Dataset002_Heart Traceback (most recent call last): File "/anaconda3/envs/nn_UNet/bin/nnUNetv2_plan_and_preprocess", line 33, in <module> sys.exit(load_entry_point('nnunetv2', 'console_scripts', 'nnUNetv2_plan_and_preprocess')()) File "/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_entrypoints.py", line 182, in plan_and_preprocess_entry extract_fingerprints(args.d, args.fpe, args.npfp, args.verify_dataset_integrity, args.clean, args.verbose) File "/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 46, in extract_fingerprints extract_fingerprint_dataset(d, fingerprint_extractor_class, num_processes, check_dataset_integrity, clean, File "/nnUNet/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 29, in extract_fingerprint_dataset verify_dataset_integrity(join(nnUNet_raw, dataset_name), num_processes) File "/nnUNet/nnUNet/nnunetv2/experiment_planning/verify_dataset_integrity.py", line 167, in verify_dataset_integrity assert all(labels_present), 'not all training cases have a label file in labelsTr. Fix that. Missing: %s' % missing AssertionError: not all training cases have a label file in labelsTr. Fix that. Missing: ['la_003', 'la_004', 'la_005', 'la_007', 'la_009', 'la_010', 'la_011', 'la_014', 'la_016', 'la_017', 'la_018', 'la_019', 'la_020', 'la_021', 'la_022', 'la_023', 'la_024', 'la_026', 'la_029', 'la_030']

…

On Tue, May 2, 2023 at 9:32 AM Fabian Isensee ***@***.***> wrote: @yejin2008 <https://github.com/yejin2008> you need to use nnUNetv2_convert_MSD_dataset, not nnUNetv2_convert_old_nnUNet_dataset. The datase you downloaded is in the MSD format. This could also be the cause of your problem @NajaJean <https://github.com/NajaJean> — Reply to this email directly, view it on GitHub <#1409 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC45RPQZHOBPNQKMDNBND3DXEEEF5ANCNFSM6AAAAAAXEVNWPA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

2 replies

FabianIsensee May 3, 2023
Maintainer

Hm ok let me double check this

FabianIsensee May 3, 2023
Maintainer

It works for me:

download task2 from here: https://drive.google.com/drive/folders/1HqEgzS8BV2c7xYNrZdEAnrHk7osJJ--2
extract tar archive
nnUNetv2_convert_MSD_dataset -i Task02_Heart -overwrite_id 988 (I have to use a different dataset ID)
nnUNetv2_plan_and_preprocess -d 988

done

yejin2008 · 2023-05-03T17:57:10Z

yejin2008
May 3, 2023
Author

Thank you so much! I got new error after I followed your instructions: nnUNetv2_plan_and_preprocess -d 988 Fingerprint extraction... Dataset988_Heart Using <class 'nnunetv2.imageio.simpleitk_reader_writer.SimpleITKIO'> as reader/writer 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:04<00:00, 4.87it/s] Experiment planning... 2D U-Net configuration: {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 40, 'patch_size': array([320, 256]), 'median_image_size_in_voxels': array([320., 232.]), 'spacing': array([1.25, 1.25]), 'normalization_schemes': ['ZScoreNormalization'], 'use_mask_for_norm': [True], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': (2, 2, 2, 2, 2, 2, 2), 'n_conv_per_stage_decoder': (2, 2, 2, 2, 2, 2), 'num_pool_per_axis': [6, 6], 'pool_op_kernel_sizes': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'conv_kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'unet_max_num_features': 512, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': True} Using <class 'nnunetv2.imageio.simpleitk_reader_writer.SimpleITKIO'> as reader/writer 3D fullres U-Net configuration: {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': array([ 80, 192, 160]), 'median_image_size_in_voxels': array([115., 320., 232.]), 'spacing': array([1.37, 1.25, 1.25]), 'normalization_schemes': ['ZScoreNormalization'], 'use_mask_for_norm': [True], 'UNet_class_name': 'PlainConvUNet', 'UNet_base_num_features': 32, 'n_conv_per_stage_encoder': (2, 2, 2, 2, 2, 2), 'n_conv_per_stage_decoder': (2, 2, 2, 2, 2), 'num_pool_per_axis': [4, 5, 5], 'pool_op_kernel_sizes': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [1, 2, 2]], 'conv_kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'unet_max_num_features': 320, 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'batch_dice': False} Plans were saved to /nnUNet/dataset/nnUNet_preprocessed/Dataset988_Heart/nnUNetPlans.json Preprocessing... Traceback (most recent call last): File "/anaconda3/envs/nn_UNet/bin/nnUNetv2_plan_and_preprocess", line 33, in <module> sys.exit(load_entry_point('nnunetv2', 'console_scripts', 'nnUNetv2_plan_and_preprocess')()) File "/nnUNet/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_entrypoints.py", line 201, in plan_and_preprocess_entry preprocess(args.d, args.overwrite_plans_name, args.c, np, args.verbose) File "/nnUNet/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 127, in preprocess preprocess_dataset(d, plans_identifier, configurations, num_processes, verbose) File "/nnUNet/nnUNet/nnunetv2/experiment_planning/plan_and_preprocess_api.py", line 96, in preprocess_dataset raise RuntimeError( RuntimeError: The list provided with num_processes must either have len 1 or as many elements as there are configurations (see --help). Number of configurations: 3, length of num_processes: 2

…

On Wed, May 3, 2023 at 5:43 AM Fabian Isensee ***@***.***> wrote: It works for me: - download task2 from here: https://drive.google.com/drive/folders/1HqEgzS8BV2c7xYNrZdEAnrHk7osJJ--2 - extract tar archive - nnUNetv2_convert_MSD_dataset -i Task02_Heart -overwrite_id 988 (I have to use a different dataset ID) - nnUNetv2_plan_and_preprocess -d 988 done — Reply to this email directly, view it on GitHub <#1409 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC45RPWCPBSVIDWQPP3LWPTXEISFNANCNFSM6AAAAAAXEVNWPA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

0 replies

one2clouds · 2024-08-12T05:35:30Z

one2clouds
Aug 12, 2024

assert len(training_identifiers) == expected_num_training, 'Did not find the expected number of training cases '
AssertionError: Did not find the expected number of training cases (20). Found 1 instead.
Examples: ['l']

I got above error.
Let's focus on the json part of the code.

json_dict = {}
    json_dict['name'] = "CT fracture all 54 res"
    json_dict['tensorImageSize'] = "4D"
    json_dict['reference'] = "CT fracture all 54 res"
    json_dict['release'] = "2022.11.03"

    json_dict['labels'] = {
        "background":0,
        "Sacrum" : 1,
        "Left Hipbone": 2,
        "Right Hipbone" : 3,
    }

    json_dict['channel_names'] = {0: "CT"}

    json_dict['numTraining'] = n_case
    json_dict['numTest'] = 0
    json_dict['file_ending'] = ".mha"
    json_dict['test'] = []

    save_json(json_dict, os.path.join(out_base, "dataset.json"))

In above, I had just changed the file enging from mha format to nii.gz and it works like a charm.

    json_dict['file_ending'] = ".nii.gz"

And run the code ---> Dataset600_Abdomen_Pelvic.py <--- again.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot pass nnUNetv2_plan_and_preprocess -d 002 --verify_dataset_integrity #1409

{{title}}

Replies: 5 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Cannot pass nnUNetv2_plan_and_preprocess -d 002 --verify_dataset_integrity #1409

yejin2008 Apr 19, 2023

Replies: 5 comments · 2 replies

NajaJean Apr 24, 2023

FabianIsensee May 2, 2023 Maintainer

yejin2008 May 2, 2023 Author

FabianIsensee May 3, 2023 Maintainer

FabianIsensee May 3, 2023 Maintainer

yejin2008 May 3, 2023 Author

one2clouds Aug 12, 2024

yejin2008
Apr 19, 2023

Replies: 5 comments 2 replies

NajaJean
Apr 24, 2023

FabianIsensee
May 2, 2023
Maintainer

yejin2008
May 2, 2023
Author

FabianIsensee May 3, 2023
Maintainer

FabianIsensee May 3, 2023
Maintainer

yejin2008
May 3, 2023
Author

one2clouds
Aug 12, 2024