SMPL to CARLA pose transfer logic

[zybermonk]

Hello,

I like your project, it is very interesting.

I would like to ask for a small clarification regarding a function logic in this repo. In the following function below, I understood that this is used to convert SMPL poses (RH system) into CARLA relative pose (LH system) by performing a series of transformations and operations. Could you please help understand the theory behind this?

So far, as per my understanding, SMPL poses are relative to its T pose (zero), and CARLA has a different reference T pose (non zero), but to transfer this pose effectively, considering the joint mappings, I fail to understand the core logic.

Some detailed explaination here would be very beneficial. Especially, the lifecylce of transformations that the SMPL pose goes through and in what sequence, to finally achieve carla_rel_rot.

Edit: Also this obtained carla_rel_rot is it with respect to the reference pose of CARLA or with respect to the previous frame of the motion?

Thank you very much.

def convert_smpl_to_carla(self, smpl_pose, age, gender, reference_pose=None) -> Tuple[torch.Tensor, torch.Tensor]:
        clip_length = smpl_pose.shape[0]

        if reference_pose is None:
            reference_pose = carla_reference.get_poses(
                self.device, as_dict=True)[(age, gender)]

        local_rot = self.__get_local_rotation(clip_length, age, gender)

        # convert SMPL pose_body to changes rotation matrices
        nx_smpl_pose = SMPL_SKELETON.map_from_original(
            smpl_pose) * self.reference_axes_dir
        mapped_smpl = euler_angles_to_matrix(nx_smpl_pose, 'XYZ')

        # write changes for common joints (all SMPL except Spine2)
        ci, si = get_common_indices(SMPL_SKELETON)
        changes = eye_batch(clip_length, self.nodes_len, self.device)
        changes[:, ci] = mapped_smpl[:, si]

        # special spine handling, since SMPL has one more joint there
        changes[:, CARLA_SKELETON.crl_spine01__C.value] = torch.bmm(
            mapped_smpl[:, SMPL_SKELETON.Spine3.value],
            mapped_smpl[:, SMPL_SKELETON.Spine2.value]
        )

        # recalculate changes from the local reference system perspective
        local_changes = torch.bmm(
            torch.linalg.solve(local_rot.reshape((-1, 3, 3)),
                               changes.reshape((-1, 3, 3))),
            local_rot.reshape((-1, 3, 3))
        ).reshape((clip_length, -1, 3, 3))

        ref_carla_rel_loc, ref_carla_rel_rot = self.__get_carla_reference_relative_tensors(
            clip_length, age, gender)

        carla_abs_loc, carla_abs_rot, carla_rel_rot = reference_pose(local_changes,
                                                                     ref_carla_rel_loc,
                                                                     ref_carla_rel_rot)

        return carla_rel_rot, carla_abs_loc, carla_abs_rot