How to get cameraToWorldUnity matrix?

[luffy-yu]

Thanks for your nice work.
I'm trying to understand the whole procedure, but I can't figure out where the CameraToWorldUnity matrix comes from. I saw you put it in a text file. How do you get it?

[doughtmw]

Thanks for your interest in this project, @luffy-yu! Unfortunately, with the current MixedReality-WebRTC implementation, there is no access provided to the synced per-frame CameraToWorldUnity matrices which are required for pose estimation while a user moves around wearing the HoloLens 2. For my work, I rigidly mounted the headset and printed out the CameraToWorldUnity using an approach from another repo of mine, after which I used the computed matrix with HMD-EgoPose for drill tracking on the rigidly mounted headset.

I am sure there is a way to modify the WebRTC implementation to access the per-frame camera parameters (intrinsic and extrinsic) of the HoloLens 2, but I didn't have the time to investigate.

[luffy-yu]

@doughtmw Thanks for your detailed reply. I have dived into HoloLens 2 for a long time. I did find a solution to stream camera parameters along with the video frame. It works as expected, but it is based on socket rather than WebRTC. Besides that, its client works on Python, not C#. It should not be hard to make some adaptations.
Following your hints, I will try to stream real-time camera parameters and work out the CameraToWorldUnity matrix.

[luffy-yu]

@doughtmw I figured out streaming extrinsic parameters of the left front camera, i.e., IResearchModeCameraSensor.GetCameraExtrinsicsMatrix().

A sample is like

[[ 7.73402284e-13  7.00452394e+31  0.00000000e+00  0.00000000e+00]
 [-3.99546136e+23  4.59079390e-41  0.00000000e+00  0.00000000e+00]
 [            nan  0.00000000e+00  0.00000000e+00  3.38098567e-39]
 [ 2.25000000e+00  6.42855480e-40  0.00000000e+00  7.91733632e-43]]

You see some numbers are very large, while some are very small.

Does it make sense?

[doughtmw]

@luffy-yu awesome! The results seem to be a bit strange though, I'm wondering if there's an error in the way you're debugging the matrix params (float vs double for example)? In this repo, the CameraToWorldUnity matrix parameters were:

CameraToWorldUnity: 
0.99998	    0.00714	    -0.00042    0.00184
-0.00712    0.99973	    0.02223	    0.12197
-0.00058    0.02223	    -0.99975    0.13058
0.00000	    0.00000	    0.00000	    1.00000

[luffy-yu]

@doughtmw I gave up the solution of getting extrinsic parameters via IResearchModeCameraSensor.GetCameraExtrinsicsMatrix(). Instead, I refer to locatable camera to get the camera's position and rotation in the world coordinate.