BlazePose 3D Coordinate mapping Problem

[tolotratlt]

Hello,

I've been working on a project in Unity that involves utilizing TensorFlow Lite's BlazePose model to detect and map the 3D coordinates of human body joints to a 3D Mecanim humanoid. While I've managed to get the model to detect the joints and return the coordinates, I've been experiencing issues with the mapping itself.

The problem seems to be that the coordinates I'm obtaining from the BlazePose model aren't working correctly when I try to map them to the 3D Mecanim humanoid. I'm not sure if this is a coordinate issue or if there's another problem, such as an orientation issue.

I've tried to troubleshoot the issue by comparing the coordinates I'm obtaining from the BlazePose model to the actual coordinates of the Mecanim humanoid, but so far I haven't been able to identify any discrepancies.

Additionally, I've just attempted a simple test where I create a direction vector from the shoulder 3D coordinate to the elbow 3D coordinate and try to apply it to the 3D Mecanim avatar. However, the avatar behaves strangely, which suggests that there may be a fundamental problem with the coordinate system or orientation.

I would like to clarify that I am not referring to any stuttering issues, but rather a possible orientation issue that is affecting the behavior of the avatar.

If anyone has any insights or suggestions on how to resolve this issue, I would greatly appreciate it.
Thank you in advance for your help!

[asus4]

@tolotratlt do you have a capture video?

MediaPipe provides 2 types of keypoints: pose_landmarks and pose_world_landmarks. In the Unity example

tf-lite-unity-sample/Assets/Samples/BlazePose/PoseLandmarkDetect.cs

Lines 20 to 21 in e0f4bed

	public Vector4[] viewportLandmarks;
	public Vector4[] worldLandmarks;

If you are using viewportLandmarks, please try to use worldLandmarks, which might have a more accurate z coordinate kye-point.

Please refer to the Here's the original document in the MediaPipe repo.

I am unfamiliar with avatar systems using body tracking, but it might be better to use a system like IK rather than mapping 3D coordinates directly.