Backbone training: Unclear requirements

[LadnerJonas]

After setting up the 160224_haggling1 data folder, I wanted to run:
python -u tools/train_3d.py --cfg configs/panoptic_ssl/resnet50/backbone_pseudo_hrnet_soft_9videos.yaml

Sadly, this does not work. Given the output of the command:

[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./data/panoptic-toolbox/data/160226_haggling1/hdImgs/00_03/00_03_00008391.jpg'): can't open/read file: check file path/integrity
...

It seems like all the data folders of the following list have to be set up before the pre-trained models can be used:

#./lib/dataset/panoptic(_ssv).py
TRAIN_LIST = [
    "160422_ultimatum1",
    "160224_haggling1",
    "160226_haggling1",
    "161202_haggling1",
    "160906_ian1",
    "160906_ian2",
    "160906_ian3",
    "160906_band1",
    "160906_band2",
    #"160906_band3",
]

Is this assumption correct?
Given that is quite resource-heavy to store 200+gb of files before we can even test anything, can you please share a minimal working example? E.g. only using the 160224_haggling1? This would make this project more attractive for others to experiment with.

[LadnerJonas]

Also, a small section on how to use this project with your videos and data would greatly help. What are the requirements? How to extract information for the config files? etc.

[keqizero]

Hi,

Thank you for the details. I believe the simplest way to conduct a small experiment is to comment the video names that you don't have yet, for both TRAIN_LIST and VAL_LIST in ./lib/dataset/panoptic(_ssv).py.

For instance:

TRAIN_LIST = ["160224_haggling1"]
VAL_LIST = ["160906_pizza1"]

You can also set VAL_LIST as the same video if you wish. We will add this in README file for quicker experiments.

For custom dataset, apart from the pseudo 2d label generation (you may choose any 2d pose detector), the key configurations are the 3d space size and space center coordinates. Please check microsoft/voxelpose-pytorch#10. However, we don't have relevant code as we haven't done experiments with other datasets ourselves. Sorry.

[LadnerJonas]

Thank you for your reply.

I did setup the TRAIN/VAL_LIST like in your example and downloaded both datasets. It did setup all the necessary folder structures as described in the README file and extracted all information from them. Nonetheless, I am pretty stuck on this error:

[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./data/panoptic-toolbox/data/160422_ultimatum1/hdImgs/00_23/00_23_00014387.jpg'): can't open/read file: check file path/integrity
...



Traceback (most recent call last):
  File "/<path>/SelfPose3dParentFolder/SelfPose3d/tools/train_3d.py", line 241, in <module>
    main()
  File "/<path>/SelfPose3dParentFolder/SelfPose3d/tools/train_3d.py", line 205, in main
    train_3d(config, model, optimizer, train_loader, epoch, final_output_dir, writer_dict)
  File "/<path>/SelfPose3dParentFolder/SelfPose3d/tools/../lib/core/function.py", line 248, in train_3d
    loss_2d, heatmaps = model(
                        ^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/parallel/data_parallel.py", line 183, in forward
    return self.module(*inputs[0], **module_kwargs[0])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/SelfPose3dParentFolder/SelfPose3d/tools/../lib/models/multi_person_posenet.py", line 46, in forward
    device = all_heatmaps[0].device
             ~~~~~~~~~~~~^^^
IndexError: list index out of range

I would greatly appriciate if you could guide me towards possible solutions for this error. I do not understand why I get plenty of this warning logs of missing pictures, when 160422_ultimatum1 and other dataset are not in this training or eval list. Also I do not quite understand why all_heatmaps seems to be not setup correctly.
Thanks!

[keqizero]

Thank you for your reply.

I did setup the TRAIN/VAL_LIST like in your example and downloaded both datasets. It did setup all the necessary folder structures as described in the README file and extracted all information from them. Nonetheless, I am pretty stuck on this error:

[ WARN:[email protected]] global loadsave.cpp:248 findDecoder imread_('./data/panoptic-toolbox/data/160422_ultimatum1/hdImgs/00_23/00_23_00014387.jpg'): can't open/read file: check file path/integrity
...



Traceback (most recent call last):
  File "/<path>/SelfPose3dParentFolder/SelfPose3d/tools/train_3d.py", line 241, in <module>
    main()
  File "/<path>/SelfPose3dParentFolder/SelfPose3d/tools/train_3d.py", line 205, in main
    train_3d(config, model, optimizer, train_loader, epoch, final_output_dir, writer_dict)
  File "/<path>/SelfPose3dParentFolder/SelfPose3d/tools/../lib/core/function.py", line 248, in train_3d
    loss_2d, heatmaps = model(
                        ^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/parallel/data_parallel.py", line 183, in forward
    return self.module(*inputs[0], **module_kwargs[0])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/.local/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<path>/SelfPose3dParentFolder/SelfPose3d/tools/../lib/models/multi_person_posenet.py", line 46, in forward
    device = all_heatmaps[0].device
             ~~~~~~~~~~~~^^^
IndexError: list index out of range

I would greatly appriciate if you could guide me towards possible solutions for this error. I do not understand why I get plenty of this warning logs of missing pictures, when 160422_ultimatum1 and other dataset are not in this training or eval list. Also I do not quite understand why all_heatmaps seems to be not setup correctly. Thanks!

Sorry I forgot one thing. Our current code is directly reading the detected 2d poses from the pkl file you downloaded. That means, the dataset content is already decided.

The modification of the TRAIN_LIST or VAL_LIST will only affect the original VoxelPose training, as it will generate corresponding pkl when the pkl doesn't exist (by 3d-2d projection).

The problem can be fixed by directly changing the content of the pkl file, or by changing the len and getitem function in the JointsDatasetSSV.py.

Sorry that I currently have some trouble accessing the data. I will add the function for subset training once I have chance. Thank you!