Generation stuck

[mingqiJ]

Hello!
I have tried the code to generate the Physion dataset(because the origin dataset doesn't contain depth image)
But the code didn't work.

I use the 1.8.8.0 version of TDW.
It is stuck at 50 line of python3.10/site-packages/tdw/controller.py: self.socket.recv()

[felixbinder]

Please try using a more recent version of TDW, eg v1.11.

[felixbinder]

Let me know if that works. If it doesn't, I'll try to sort out the specific versions that we used back in the day.

[mingqiJ]

Hi,
Thanks for your reply. I have tried 1.11. This is the version I used

tdw                       1.11.23.0
tdw-physics               0.3.1

But it still didn't work. The same problem: It is stuck at 72 line of python3.10/site-packages/tdw/controller.py: self.socket.recv().
No matter how long I wait, it still didn't work.

I also checked the system, I can see the python process and TDW process. The %cpu of python is 0, it looks like it is waiting.

[mingqiJ]

Hi @felixbinder , I have fixed this issue by setting launch_build=False and launch TDW by myself.

But it takes error if I use 1.11.23 version TDW:(It is in 1009 line of controllers/dominoes.py)

target_position_new = self.get_object_position(self.target_id, resp) or self.target_position
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Only 1.8.8.0 version of TDW can work.

Hope this information is useful

[felixbinder]

Awesome, I'm glad you figured it out! Seems to be something to do with the interprocess communication—that can be hard to figure out. Thanks for the update

[mingqiJ]

Hi @felixbinder, it still didn't work for dataset generation:

I used 1.8.8.0 version of TDW
When I directly use the dominoes.py, it can run and generate the video.
python controllers/dominoes.py --port 1071 --dir try

But if I use the command the generate dataset, it stuck at the same position....
python3 controllers/dominoes.py @./configs/dominoes/pilot_dominoes_0mid_d3chairs_o1plants_tdwroom/commandline_args.txt --dir try_2 --height 256 --width 256 --seed 0 --save_passes '' --write_passes '_img,_depth,_id' --save_meshes --num_multiplier 7.25 --training_data_mode --gpu 0 --port 1071
Stuck here in tdw/controller.py:
resp = self.socket.recv_multipart()

pilot_dominoes_0mid_d3chairs_o1plants_tdwroom didn't work
pilot_dominoes_1mid_J025R45_boxroom work

Do you know why?

[felixbinder]

One possibility might be that you do not have the right models installed. If I remember correctly, some of the models of the distractor/occluder objects that are placed in front/in the background are drawn from a library of 3D models that is only included by request in tdw. To test this, you could modify the generation code to not include any distractor/occluder objects and see if it still hangs. If that is the case, it is probably the proprietary model library.

To get access to those models, write Jeremy Schwartz an email and cc me ([email protected])

[mingqiJ]

Hi, thanks for your help.

Yes. if don't set any distractors and occluders, the code can work for other 7 scenarios(without Drape). Drape needs flex.json file. I have tried all of them.

If include distractor and occluder, the code will stuck in tdw/controller.py, and wait all the time: resp = self.socket.recv_multipart().

[mingqiJ]

Hi @felixbinder,

I have successfully set it up . And the Generation code can work.
I didn't try all of them. But some ones didn't work before can work now. I am really appreciate your help.

I also have another question about the depth image process.

Do you have any instructor about how to transfer the depth image to point cloud.
I thought a good guide for this is very useful.

Because I have used camera_matric and projection_matric, but the depth point cloud is still not exactly consistent with true point cloud(3d position). So I think I loss some detail parameters.

Here is some detail problems:

1. What is near_plane and far_plane for Physion dataset? I need these to get the true depth value.
2. Is the depth value you saved distance between object and camera or distance between object and camera plane?
3. If we get the true setup, how should we do the process for the depth image at image coordinate? Like firstly normalize and then add camera matrics? How to normalize, just / resolution?

[felixbinder]

Hello, I'll loop in @htung0101 , who was working on the point cloud model pipeline. I'm personally not familiar with how that part of the pipeline works.

[mingqiJ]

Hi @htung0101 , do you have any guidance on this? It is vey important to know how to convert the Depth into Point Cloud. I would like to know how you set it up, because now I can't match the transformed point cloud with the real world coordinates exactly. Thanks.