Are Guidance Losses used in training?

[HeimbergM]

Hello! Thanks for this great work!

I am trying to use ScoreHMR in a multi-view setting where camera calibration is known.

I already added the following changes: I made multi-view setting batch-able.
I combined keypoint loss and multiview loss.
These 2 additions increased the performance on my data.

I also implemented a cross-view keypoint loss, where i transform the predictions to a global coordinate frame and calculate the loss with the keypoints from the other perspectives. This sadly did not lead to any improvements.

After reading the paper I wondered if the model is trained with any of the guidance losses enabled or are all of them introduced after training? Would it be possible that training with my new guidance loss could lead to better performance?

Have you thought about a way to incorporate calibration knowledge into the model if available?

[statho]

Hello! The model does not use any additional loss during training. The guidance losses are applied at test-time.

In a multi-view setup with calibrated cameras, you could triangulate 2D keypoints and obtain 3D keypoints. Then you could fit the SMPL model to the 3D keypoints.