Scaling in the Eye Gaze Estimation

[saeidn95]

It's about scaling for eyeSize in (

human/src/face/angles.ts

Line 20 in 6239631

const eyeDiff: Point = [ // x distance between extreme point and center point normalized with eye size

)

The iris has two rotational degrees of freedom, measured by the in the vertical and and horizontal angles. The scaling suggests that what rotational angular values relative to the horizontal and vertical openings of the eye. What is the justification for that? Is there a reference that you can point me to?

[vladmandic]

sorry, there is no "reference", its my code that i wrote as i found to be best suited - if you have better suggestions, please do contribute!

[saeidn95]

Here is what I think would work. I like to have your opinion before I crack the code. The solutions depends on the accuracy of the z values from the mesh.

All the points that we use in the calculateGaze (https://github.com/vladmandic/human/blob/62396317f5c31b2ec5bc7de2e9eb53b8978b9f28/src/face/angles.ts#L5C7-L5C20) except the iris center, have three dimensions x, y, and z from mesh.

Assuming the eye ball to be a sphere, with 4 mesh points that are used in calculateGaze, we can solve a MSE to find the center and radius of the eyeball sphere. With that we can compute the z, y, z coordinates of the eye center along two perpendicular great circles on the sphere. Further, knowing the x and y coordinate of the iris center from the attention network, we can obtain its z coordinate. With z coordinate at hand it is fairly easy to use the spherical trigonometry to find two vertical and horizontal angles (or bearing and strength) of the gaze. I suspect the results would not too far from the current technique, provided drop the scaling. The way I think of it from what know of geometry is that the scaling should be dropped. Of course, I might be missing something that you had in mind to apply the scaling.

[vladmandic]

the conceptual problem with that is that facemesh xyz points are not actual predictions, they are transforms of the predefined "perfect face". basically, there is a facemesh and then based on detection points it is streched/transformed to fit those points - so z-axis precision is pretty much "yeah, it fits good-enough", its not actual prediction.

but sure, why not try - i like the idea.

[saeidn95]

Yes, I see the problem; basically it entailed doing a best squares fit on sphere on points that are are themselves are best fit afine-tranformation (translation and rotation) of the perfect face. However, we do that even now with x and y axis. If we trust the z axis as much as x and y axes, then we should be ok. I have studies this for a little bit, and in my view short of using a CNN, I think what I am proposing is best than what I have seen in the literature.

There is one paper (https://www.cl.cam.ac.uk/research/rainbow/projects/eyetab/files/eww23_etra2014.pdf) that spends a great deal of time to get the points around the limbus to build an ellipse (a projection of sphere onto a plain). Next it does a 2D to 3D projection to find the sphere and the gaze). Here we already have everything in 3D.

I will work on the problem and let you know.

[saeidn95]

Sure! If I find something useful, I will report back.