WVN results on our data

[AfonsoEloy]

Hello again guys! We have now integrated your system with our data, but the results don't quite look like the ones from the jackal sim or the paper. As such, I'm attaching some images and debug info in the hope you can check if this is normal for some reason, or perhaps point us in the right direction to tune the configs.

The relevant part of our setup for this work consists of a Unitree Go1 EDU robot and a RealSense D435i recording RGB at 640x480@6fps. The robot generates velocity estimates at 50Hz and we compute the Twist commands sent to the robot from the joystick feedback and the known maximum velocities achievable by the robot in its current gait.

First few seconds of operation with 120 images, 6 model updates and 15 valid nodes (Image callback: front... step: 80 | loss: 0.240516 | loss_trav: 0.204203 | loss_reco: 0.474659):

215 seconds of operation with 1266 images, 177 model updates and 419 valid nodes (Image callback: front... step: 1640 | loss: 0.036430 | loss_trav: 0.142824 | loss_reco: 0.068188):

240 seconds (just to show that it learned the sidewalk quite fast):

355 seconds of operation with 2117 images, 295 model updates and 768 valid nodes (Image callback: front... step: 2240 | loss: 0.048374 | loss_trav: 0.190312 | loss_reco: 0.094069)

The results then tend to stabilize, i.e., the horizon line (or canopies) roughly separate the traversable from the non-traversable regions, in a total of 1099 seconds of data.

[mmattamala]

Hi @AfonsoEloy, it's great to see you testing WVN on your data.
I suspect that something might be off with the supervision signal. A few things to check:

• Are the RGB images you feed into WVN in the right frame? In the past we have some problems with cameras that published images in the "camera frame" instead of the "optical frame". Then, the reprojected path was not working as intended.
• Are all the other frames also correct? Particularly the footprint is critical.
• Lastly, you can check the debug messages, if you change the mode using this flag. That should populate some additional images and the supervision path marker to confirm it works as intended.

Let me know how it goes

[AfonsoEloy]

Hi @mmattamala! I am glad to tell you that the problem was indeed the footprint frame -- considering that in the simulation example you were using the base_link for both the base and the footprint, I also did the same on our data. When I properly created a footprint frame for our robot, the behavior of your technique is now outstanding!

For future reference, I leave here a couple of prints in the same timestamps as before to show the improvement.

First few seconds:

240 seconds:

Once again, thank you so much for your help!

[mmattamala]

Hi @AfonsoEloy, this is awesome! I'm glad to know you discovered the problem.
Probably we should have made more clear that the footprint is critical, and some ways to define it. I believe that for the ANYmal we got that frame "for free" from the TF tree, and it is set as the projection of the 4 feet at some height from the base. For the jackal example, the base_link is defined at low height, so it made no significant difference.

Thanks a lot for sharing your results, this looks super exciting :)