Issue with gripper pose position in relation to associated calibrated cameras

[QazyBi]

Firstly, I'd like to express my gratitude for making the DROID dataset and associated code available to the open-source community.

While analyzing the Raw dataset, I observed a potential discrepancy between the gripper pose positions and the associated calibrated cameras. Below, I have included the preprocessing code for both the gripper position and camera extrinsics. Additionally, I've attached point-cloud images from one episode viewed from various angles, where the yellow indicates the camera, green denotes the current gripper position, and red marks the next gripper state change position:

# Preprocessing of the gripper position
robot_pose = np.zeros(8)
dataset_robot_pose = trajectory['action']['cartesian_position'][t_index]
robot_pose[:3] = dataset_robot_pose[:3]
robot_pose[3:7] = R.from_euler("xyz", dataset_robot_pose[3:]).as_quat()
robot_pose[7] = False

# Preprocessing of the extrinsics
pos = extrinsics[0:3]
rot_mat = R.from_euler("xyz", extrinsics[3:6]).as_matrix()

extrinsics = np.zeros((4, 4))
extrinsics[:3,:3] = rot_mat
extrinsics[:3,3] = pos
extrinsics[3,3] = 1.0
extrinsics = np.linalg.inv(extrinsics)

Could you please review this? Any guidance on whether this is an error in the dataset or a misunderstanding on my part would be greatly appreciated.

[kylesargent]

Did you figure it out?

[QazyBi]

forgot to do this when doing a visualization:

camera_pose = np.linalg.inv(extrinsics)

[arsalan-mousavian]

@QazyBi where did you extract extrinsics and intrinsics from? I can't find it in the sample DROID_100 data.

[QazyBi]

@arsalan-mousavian, I used raw dataset. For anyone who's willing to visualize the dataset I suggest looking for this repository: https://github.com/rerun-io/python-example-droid-dataset/tree/master

[QazyBi]

@arsalan-mousavian, refer to this issue for more details on camera parameters #4