Integrating ProtoMotions with PHC Retargeting Output

[IlanElfen]

Hi, thank you for this great work

I've ran PHC's retargeting pipeline based on the AMAS dataset on a custom humanoid robot. The output for each motion is .pkl file, which differs from the expected .npy or .yaml files. How can I change my PHC output motion files to resemble the ProtoMotions input motion file?

I understand that there is a pipeline that converts motions to the Unitree H1 configuration but can't find where the retargeting happens. How was this done (what is the H1 motion pipeline), and is it possible to do with a custom robot?

Thanks!

[tesslerc]

See the README for the data pipeline for the ProtoMotions codebase.
It's very similar to PHC, but has some small differences.

The code is relatively versatile and can easily support new robots. You just need to create a corresponding configuration file under phys_anim/config/robot/ and make sure your data matches the required format.

You can see the H1 retargeting code (that is adapted from PHC/H2O) here. We save the full-body translation, rotation, and velocities tensors. They are then loaded here:

ProtoMotions/phys_anim/utils/motion_lib_h1.py

Line 80 in f97296d

def _load_motion_file(motion_file):

The difference with the standard AMASS to SMPL/X pipeline is that those are saved in PoseLib format and also include the kinematic skeleton.

[IlanElfen]

Got it, thanks. Just to clarify, I have to re-generate all the motions using ProtoMotions, right? I can't use the ones already given by PHC.
I'm also confused how to define the rotation_axis is robot_humanoid_batch.py. What is the correct order of the rotation_axis, and should I also define fixed joints as [0, 0, 0]? For H1, the order of the rotation_axis corresponds with the original_body_names, but since I don't have the same number of bodies as joints, I struggle to understand how to populate this array correctly. Thank you!

[tesslerc]

You can convert between PHC and ProtoMotions, but we don't provide such a script.
If you create one, I'd be happy to integrate it into the codebase for others to use.

For the H1 retargeting script, I have uploaded a new version based on what PHC provides in the H1 branch https://github.com/zhengyiluo/phc/tree/h1_phc .
The retargeting script has been slightly altered to return the joints in the proper ordering and to fix heights.

Keep in mind the H1 is now updated with better naming. The extended_.* convention is confusing. Now the default H1 is with head and hands, and you can choose h1_no_head_no_hands or only _no_head to remove specific body parts from the observation.

Please let me know if you encounter any issues.

[IlanElfen]

I got the motion retargeted with your updated script.
Thank you for your help!

[IlanElfen]

Sorry to reopen this issue, the retargeting didn't work.
Fitting the robot's shape did but fitting the motions didn't, specifically convert_to_proper_kinemtaics in torch_humanoid_batch.py gave me an Index Out of Bounds:
IndexError: index 76 is out of bounds for dimension 0 with size 76

I have 78 bodies and 76 joints (not all end effectors have joints), which is probably causing the issue.
These commands ran without problem:
return_dict.global_translation = ...
return_dict.global_velocity = ...

While these return the error above:
return_dict.global_rotation_mat = ...,
return_dict.global_rotation = ...,
return_dict.global_angular_velocity = ...

Removing the line that calls that convert_to_proper_kinematic makes the whole program run, so the issue is only due to that function.
This function isn't in PHC so I'm raising the issue here.
Do you know what could cause this issue? Thanks!

[tesslerc]

What commandline did you run? I'll try to reproduce on my end and see.

PHC/H2O use an adhoc extended_* method. They need to extend their skeleton in code.
We adapted it here to return the proper kinematic ordering. And then leverage the proper XML/URDF functionality to get the real coordinates and kinematic structure with the end-effectors (head/hands) in simulation.

convert_to_proper_kinematic simply re-orders the joints from the extended format to the expected ordering based on DFS skeleton tree traversal.

[IlanElfen]

I ran python convert_h1_to_isaac.py with my robot as the default humanoid_type.
This means that if I'm not extending any end-effectors, I shouldn't need to run convert_to_proper_kinematic?

[tesslerc]

The extend config is defined in https://github.com/NVlabs/ProtoMotions/blob/main/data/scripts/retargeting/config.py
So if you extend a body part, it will expect it in the 'convert_to_proper'. If the extend config is set to an empty list, the conversion shouldn't change anything.

[IlanElfen]

Ok understood, thank you.
One more issue with the retargeting pipeline is that I'm unable to visualizing the results both from my own robot and h1.
I'm running python play_motion.py robot isaacgym {motion_file.npy} (with a motion file generated by convert_h1_to_isaac) and get the following error:
RecursionError: maximum recursion depth exceeded in comparison
I'm also get errors when running the visualization with the demo files so I can't compare the outputs to see what's going wrong.
When running:
python play_motion.py {...}/ProtoMotions/phys_anim/data/motions/h1_punch.npy isaacgym h1, an IsaacGym window pops up, and correctly loads the experiment but crashes a few seconds later with the following error:

def quat_mul(a, b, w_last: bool):
    assert a.shape == b.shape
    ~~~~~~~~~~~~~~~~~~~~~~~~~ <--- HERE
    shape = a.shape
    a = a.reshape(-1, 4)
RuntimeError: AssertionError: 

Are you able to reproduce this, or is something wrong with my configuration?

EDIT: Order of args for play_motion

[tesslerc]

Could you try play_motion with the following input order motion_file then backbone and finally robot?

[IlanElfen]

Yes, the issue comes from running the commands in that order. I wrote it wrong before, sorry.

[IlanElfen]

I reverted to a commit before the PHC parts (and the change in robot names/configs) and was able to run play_motion with h1_extended_hands and the demo motion file. Putting my own robot and motion file still resulted in the

  File "home/mambaforge/envs/protomotions/lib/python3.8/abc.py", line 98, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison

[tesslerc]

I haven't seen this in the past.
I'd like to figure out why this happens.

Could you look at the call-stack to see where the recursion is coming from?

I can also take a look if you're willing to share your robot+motion files.

[IlanElfen]

Here are my files. There's the xml file (mtb_3_0.xml), the folder with its corresponding meshes (library), the retargeting config.py
, my robot yaml, and a generated motion file ("accident... .npy").
Thank you for taking a look

[IlanElfen]

The message error starts with a hydra configuration error, which then goes back and forth between two lines before exiting with the RecursionError.
I'm not sure why this error would happen with my motion file and not the demo one. Have you been able to replicate this issue or is my local configuration wrong?

Setting GYM_USD_PLUG_INFO_PATH to /home/ilane/Downloads/IsaacGym_Preview_4_Package/isaacgym/python/isaacgym/_bindings/linux-x86_64/usd/plugInfo.json
2024-11-19 17:22:35,850 - INFO - logger - logger initialized
phys_anim/eval_agent.py:66: UserWarning: 
The version_base parameter is not specified.
Please specify a compatability version level, or None.
Will assume defaults for version 1.1
  @hydra.main(config_path="config")
Traceback (most recent call last):
  File "/home/ilane/mambaforge/envs/protomotions/lib/python3.8/site-packages/hydra/_internal/utils.py", line 220, in run_and_report
    ...
  File "/home/ilane/mambaforge/envs/protomotions/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 573, in _create_defaults_tree_impl
    add_child(children, new_root)
  File "/home/ilane/mambaforge/envs/protomotions/lib/python3.8/site-packages/hydra/_internal/defaults_list.py", line 520, in add_child
    subtree_ = _create_defaults_tree_impl(

The last two lines (...line 575, ...line 520) repeat until the RecursionError

[tesslerc]

Thanks for sharing.
I took a quick look, here's some pointers:

1. The defaults was set to the same file, so hydra enters a recursion error there. You want your robot config to inherit from the base class.

defaults:
  - base

2. Then you want to set the other parameters in the config:

  dfs_dof_names: 
  dof_obs_size: 
  number_of_actions: 
  key_bodies: ["base_link"]

The first 3 are definitions of the robot. You can obtain them once the robot is loaded. Printing at phys_anim/envs/humanoid/isaacgym.py in create_envs you can call

body_names = self.gym.get_asset_rigid_body_names(humanoid_asset)
dof_names = self.gym.get_asset_dof_names(humanoid_asset)
num_dof = self.gym.get_asset_dof_count(humanoid_asset)
num_joints = self.gym.get_asset_joint_count(humanoid_asset)

The num_dof is the number of actions. The dof_names is the dfs_dof_names and dof_obs_size is dof_body_ids*6 if I'm not mistaken.

We will create a script to help automate creating the robot config file in the future.

[IlanElfen]

Yes, thank you for your answer! That fixed the RecursionError and after a little more debugging, I have motion visualization

One last question: when running the same configuration (same robot, xml, matching joints, etc) in PHC and Protomotions, the retargeting gives different outputs:
The shape fitting is identical (ProtoMotions - Left, PHC - Right):

And so is the motion fitting for the motion file:

However, the retargeting is different, as you can see:

Screencast.from.11-21-2024.02.41.50.PM.webm

There's no configuration difference, so I'm struggling to find what's causing this difference.
As always, thank you very much for your help!

[tesslerc]

If I'm seeing correctly, the red spheres seem to be performing the right motion.
But the rotations when initializing the dofs are wrong. Is there an extended joint that needs to be handled properly in the retargeting joint ordering?

I tried to bring it up on my end, but I think there's a mismatch between the motion and the model. From what I see, the motion returns 76 dof but the loaded model is 75?

Happy to hop on a call next week to help solve this, or if you're at SIGGRAPH Asia we can debug this together.

[IlanElfen]

Sorry for the late response. The dof mismatch was a previous issue I had, which I resolved. The Drive files have been updated if you're still willing to take a look. I'm not extending any dofs.
I believe you're coming to Mentee in the coming days, where I work. If the issue isn't resolved then, can we debug it together there? Thanks

[tesslerc]

Thanks, I can now run it on my end and I see the same results as you do on ProtoMotions.

Could you run the retargeting script in PHC (what worked for you when replaying in PHC codebase), but save in the format expected by ProtoMotions?

Change this line: https://github.com/ZhengyiLuo/PHC/blob/656023e984ea84d2e2a51352049ff99f9cc67c4b/scripts/data_process/fit_smpl_motion.py#L174
To match this:

ProtoMotions/data/scripts/retargeting/fit_smpl_motion.py

Line 216 in ed21d63

curr_motion = {

It will help me know where to focus.

[IlanElfen]

I did that and get the same results as when using ProtoMotions (robot motions don't follow the correct retargeting motions).
I had to add this before changing the line you suggested:

fk_return = humanoid_fk.fk_batch(
            pose_aa_h1_new.detach(),
            (root_trans_offset_dump[None,] + root_pos_offset).detach(),
            return_full=True,)

Without it, the dict was missing all relevant entries and it wouldn't visualize.