Proposal for recenter issue with the daydream HMD

[Tivoyagefeak]

Hi, first of all thank you for your great work. Im not sure if I overthought the following problem, but I had an issue with the build in recenter Action from DayDream. The behaviour of the ARCam in relation to the VR Camera really confused me and I couldnt find a solution anywhere. So If anyone stumbles across the same issue this is how I tackled it:

1. The problem:
   Everytime you recenter in a different orientation than the starting orientation of the app, the VR Camera gets aligned correctly but the ARCore Camera is not affected (which is quite obvious). But it had no affect either, when I tried to rotate the ARCore Camera to the new orientation. And I tried a lot of approaches to fix the rotation issue.
2. My solution:
   In the end I watched the last X rotations of the VR camera and listened to the GvrControllerInput.Recentered event to get noticed when the event started. You have to listen to the last rotations because it might take one or two more frames till the recenter actually takes place. Then you can spot the desired value. This angle is used to rotate the position vectors in the followARCoreCamera so you still walk straight on.

public class RecenterCorrection : MonoBehaviour {

    public static float ANGLETHRESHOLD = 2f;
    public static int ROTATIONQUEUESIZE = 10;

    //camera for rotation values
    public Transform gearCamera;
    //AR Core Camera for real world position values
    public Transform arcoreCamera;
    //class which "followy" the arcore cam position with the daydream camera
    followARCoreCamera arcorecam;
    //stores rotation each frame from daydream camera
    Queue<Vector3> rotations;
    //flag which indicates that recentered got triggered
    bool watchRotation;
    // Use this for initialization
    void Start () {
        arcorecam = GameObject.FindObjectOfType<followARCoreCamera>();
        watchRotation = false;
        rotations = new Queue<Vector3>();
    }

    // Update is called once per frame
    void Update () {

        //only watch last ROTATIONQUEUESIZE rotations. The rotation recenter can last some frames, so we have to track more rotation values to spot the action and the angle
        if (rotations.Count > ROTATIONQUEUESIZE)
         {
            //if queue is full kick out the oldest element
             rotations.Dequeue();
         }
        //enque the newest rotation value of daydream cam
         rotations.Enqueue(gearCamera.rotation.eulerAngles);

        //recentered got pressed might be executing soon / executed
        if (GvrControllerInput.Recentered)
        {
            Debug.Log("[IO] Recentering Done");
            //watch the Queue with stored rotations to spot the angle difference
            watchRotation = true;
        }

        if (watchRotation)
        {
            //get current y angle of daydream camera
            float yAngle = gearCamera.rotation.eulerAngles.y;
            //check if its inside our "recetered" anglethreshhold
            if ((yAngle > 360f - ANGLETHRESHOLD && yAngle <= 360) ||
                    (yAngle >= 0f && yAngle < ANGLETHRESHOLD)){
                //get the odlest y angle of queue
                yAngle = rotations.Dequeue().y;
                //remeber the angle before yAngle
                float angleBeforeRecenter = 0f;
                //as long ad the yAngle is not in range, dequeue the rotations
                while (!((yAngle > 360f - ANGLETHRESHOLD && yAngle <= 360) || (yAngle >= 0f && yAngle < ANGLETHRESHOLD)))
                {
                    angleBeforeRecenter = yAngle;
                    yAngle = rotations.Dequeue().y;
                }
                //now angleBeforeRecenter has the angle before the recenterd rotation of (almost) 0
                Debug.Log("[IO] Recenter finished, y Angle almost 0f");
                //tell arcore cam the angle, so the delta Vectors can be rotated accordingly
                //we have to substract the old vector, cause this vector is always the new origin rotation
                arcorecam.mirrorAngle = (arcorecam.mirrorAngle - angleBeforeRecenter)%360;
                Debug.Log("[IO] ARCore mirrorAngle = " + arcorecam.mirrorAngle);
                //stop watching the rotations
                watchRotation = false;
            }
            //if not, keep on recording the rotations of daydream cam
            rotations.Enqueue(gearCamera.rotation.eulerAngles);

        }
    }
}

If you have a more convinient solution or if i missed sth please share your thoughts. I just wanted to share this so if another person has similiar issues he can start with my approach.

Greetings

Micha

[ChristophGeske]

Hi sorry for the late response.

I use the following code which works well for the GearVR version and is a super short solution.

public class AlignementCorrection : MonoBehaviour {
public Transform arCoreCamera;
public Transform userCamera;

// Update is called once per frame
void Update () {
    // Use controller trigger or touchpad to activate realignment when headset orientation and ARCore Orientation became missaligned. 
    if (OVRInput.GetDown(OVRInput.Button.One)) 
    {
        //Unity engine XR allows to acesses the headset rotation (CenterEye part of the UserCamera object can be used to acess the rotation). The UserCamera rotation needs to be subtrackted by using the Inverse function otherwise the rotation of the ARCoreCamere and UserCamera would be addet together. We also have to set the ArCoreCamera to track position and rotation so we can use the rotation of the ArCoreCamera to reset the UserCamera to this rotation.
        userCamera.rotation = arCoreCamera.rotation * Quaternion.Inverse(UnityEngine.XR.InputTracking.GetLocalRotation(UnityEngine.XR.XRNode.CenterEye));
    }
}
}

I tried to test your code but the .mirrorAngle function gave me an error. I might have to test it again with the Daydream version maybe some google related functions are missing in the GearVR version.
I haven't thought to much about a Daydream solution for the issue so it is great you came up with a solution that works. If you like you can implement it and send me an pull request. Or I try to implement it myself if I have time. I haven't completely understood your code yet I might just need some more time to think it through. Looks very interesting. :D

Greetings
Christoph

[Tivoyagefeak]

Hey, no problem.
Yeah, you have to add a public float value to your followARCoreCamera.cs called mirrorAngle. Default value is 0, because when the app starts I assume the cameras are both aligned properly. In the followARCoreCamera Script I take the mirrorAngle and Rotate my Translation Vector

 //correct the delta vector by given angle
   if(mirrorAngle != 0f)
   {
       deltaPosition = Quaternion.AngleAxis(mirrorAngle, Vector3.up) * deltaPosition;
   }

You can ask if something is unclear. :-) If I have time i can try to implement it.

Greetings

Micha

[ChristophGeske]

Hi Micha,
I found some time to implement your suggestion and added your code to the latest update. The Cardboard/Daydream project now contains a new class called AlignmentCorrection which is basically your RecenterCorrection class with some additional code for me to test if it works, since I do not own a daydream controller and have to use display touch for the user input instead. I also added the mirrorAngle variable to the FollowARCoreCamera class like you said.

Unfortunately I am still not able to realign the cameras by touching the screen which should simulate the press of the recenter button you used in your solution.

Any idea what went wrong? Is recentering using the daydream controller working with the latest version I uploaded and did I just make a mistake with my approach to simulate the recenter button?

[Tivoyagefeak]

Hey Christoph,
I will check it on monday and keep you updated.
Have a nice weekend.
Micha

[Tivoyagefeak]

Hey, sorry for the late answer. Your FollowARCoreCamera Class should more like this:

using UnityEngine;

/* The ArCore Camera only takes 30 pictures per second which in turn leads to a maximal tracking refreshrate of 30 fps as well since we only get
 * new data points about our position in space 30 times a second. To solve this and get a 60 fps tracking to work we use a trick. By spliting the distance 
 * data we recieve in half we can use them to acive 60 fps tracking with the disadvantage of a slight lag since the virtual world only moves half way while 
 * in reality the head moves the full distance. Playing with those values and the Lerp method might further enhance the experience by reducing the lag. This needs
 * some trial and error to find the perfect settings.
 */
public class FollowARCoreCamera : MonoBehaviour {

	public Transform aRCoreCamera;

    Vector3 smoothedPosition; 
    private Vector3 startPosition;
    private Vector3 endPosition;
    private float speed = 7.5f; //2.5 works also with some slow down
    private float maxSpeed;

    // 0.0159 is the distance the camera is away from the head. This leads to position changes when turning the had and we wanna avoid that. The current setting need still more refinement. 
    private Vector3 headRotationCorrection = new Vector3(0,0,0.0159f);


    public float mirrorAngle = 0;
    private Vector3 deltaPosition;

    // the update method is twice as fast as the ARCore Camera. To avoid jittering we use Lerp to smooth out the movement at the start and the end of the movement.
    void Update () {

        startPosition = this.transform.position;
        endPosition = aRCoreCamera.transform.position - headRotationCorrection; // TODO not sure if headRotationCorrection even helps.
        maxSpeed = speed * Time.deltaTime;

        //correct the delta vector by given angle
        if (mirrorAngle != 0f)
        {
            endPosition = Quaternion.AngleAxis(mirrorAngle, Vector3.up) * endPosition;
        }

        
        // Using Lerp this way is a trick. TODO: There are probably better solutions which should also be tested and might reduce lag further.
        smoothedPosition = Vector3.Lerp(startPosition, endPosition, maxSpeed ); 
        
        this.transform.position = smoothedPosition;                             
    }

}

The translation-vector needs the corrected rotation after the recenter. I implemented it like that and it worked on daydream. The GearCamera should be the real camera object with the camera component on it (in your boxy room scene: "Camera") and the ARCore Camera object should be the "ARCore Device" (with the ARCoreSession component). After making this changes everything worked as intended.

Greetings

Micha

Hey I just looked it up in our new project and now nothing seems to work anymore. I'll try to fix it an dsend you an update

[Tivoyagefeak]

So it was a rotation problem in my scene, but your boxyroom scene should work with my code above. But its still a problem, if the ARCore Camera tracking gets interupted (by occluding the camera ). If the Daydream Rotation changes while the camera is occluded/lost tracking everything is messed up