Large centroid error

[jwd-sbb]

I have manually obtained ground truth values for the pose of my object. Using the same set of Color and Depth images, I used the FoundationPose model (run_demo.py) to estimate the object's pose. When comparing the pose outputs from FoundationPose to the ground truth, I observed the following errors:

Centroid error: ~15 mm
Angular error: ~2.8 degrees
While the angular error seems promising, the centroid error appears relatively large.

I have a question regarding the source of this discrepancy:

Is the pose from FoundationPose derived from any specific corner or coordinates of the object rather than its centroid?
Below, I’ve included two sample poses for comparison:

Pose from FoundationPose (cam2obj):

[[-0.9005515  -0.41394609  0.1328733  -0.03936549]
 [ 0.36925298 -0.88960445 -0.26880452  0.00987321]
 [ 0.22947536 -0.1930085   0.95398563  0.24826676]
 [ 0.          0.          0.          1.        ]]

Pose from Ground Truth (cam2pieces):

 [[-0.89243326 -0.42257113  0.15810272 -0.05506524]
 [ 0.36218652 -0.87993885 -0.30745494  0.00842595]
 [ 0.26904231 -0.21712035  0.93833628  0.24838575]
 [ 0.          0.          0.          1.        ]]

Errors Observed:

• Centroid error: 15.77 mm
• Angular error: 2.81 degrees

Could the large centroid error be explained by how the pose is being estimated (e.g., if it's calculated from a specific point like a corner rather than the centroid)?

[wenbowen123]

• how did you obtain GT?
• How does the result look like qualitatively?
• How does your GT look like qualitatively? (overlay the mesh like in our demo)

[jwd-sbb]

Hi @wenbowen123,

We have created an internal tool to obtain the GT (see the image below, 0011.png, which shows how it appears with the CAD model overlay).

Also adjusted the FP code to overlay the CAD model according to the pose (0011.png).

Here is the link where you can access both the dataset and the debug folder.
https://drive.google.com/drive/folders/1tRK9uPqvUmfDA0wUiAGcRVItMoe2Yqcz?usp=drive_link

When directly comparing the angular error between the GT tool and the FP pose, I observed a substantial angular discrepancy of 122.65 degrees. Upon further investigation of the rotation matrices from both the FP and GT tools, I found that the FP rotation matrix had a misalignment between the x, y and z axes. To correct this, I reshuffled the FP rotation matrix, as shown below, to better align it with the GT tool pose. After this adjustment, the angular error fell within an acceptable range, though the centroid error remained around 15 mm, as mentioned earlier.

pose[:, :3] = pose[:, [2, 0, 1]]

pose[:3,0] *= -1
pose[:3,2] *= -1

[wenbowen123]

if you visualize "scene_raw.ply" in the debug folder, the PCL is very bad. The cube is flat as a plane. What camera did you use? Consider using a better RGBD camera.

[wenbowen123]

your cube model also has no texture. So there is symmetry ambiguity.

[jwd-sbb]

if you visualize "scene_raw.ply" in the debug folder, the PCL is very bad. The cube is flat as a plane. What camera did you use? Consider using a better RGBD camera.

I am using the Intel D435 depth camera, https://www.intelrealsense.com/depth-camera-d435/

[jwd-sbb]

your cube model also has no texture. So there is symmetry ambiguity.

Most of the models I work with are low-textured and commonly used in automative industry. Would adjusting any parameters in FoundationPose improve its performance?

I plan to improve the lighting conditions and observe how it affects the pose estimation results.

[wenbowen123]

by no texture, I'm not saying the object itself is textureless (this is fine, and your object actually has some black, brown and gray segments), but your 3d CAD model is in purely milk white, which is quite far from what the object really is. It's better to change the color for that.

[wenbowen123]

the key issue I believe is the point cloud being too bad. You can see the contour kinda matches, but the estimated cube is a bit below the table (that's how the PCL looks like being flat).

[jwd-sbb]

by no texture, I'm not saying the object itself is textureless (this is fine, and your object actually has some black, brown and gray segments), but your 3d CAD model is in purely milk white, which is quite far from what the object really is. It's better to change the color for that.

Thanks, I will try to update the CAD model.

[jwd-sbb]

the key issue I believe is the point cloud being too bad. You can see the contour kinda matches, but the estimated cube is a bit below the table (that's how the PCL looks like being flat).

I updated the dataset with improved lighting conditions abit, while keeping the CAD model unchanged for now, and achieved better results.

To achieve this, I had to adjust the signs and reindex the rotation values.

pose[:, :3] = pose[:, [2, 0, 1]]
pose[:3,0] *= -1
pose[:3,2] *= -1

pose[0,3] *= -1

Total number of poses: 452

Number of poses those centroid error is
< 5mm: 103
> 5mm: 349
Number of pose those angular error is
< 5 deg: 336 
> 5 deg: 116 

I'll test it under improved lighting conditions, make updates to the CAD model, and provide an update soon.