World positions of left/right eye

WendelinReich1 · August 25, 2015, 3:25pm

Hi,

I need to get the precise world positions of left and right eye in Unity 5.1s native VR support (read-only). Is there an API call for this, seeing how the VR Camera rig is now constructed and handled by Unity behind the scenes? It’s easy if one uses the OVRCameraRig Prefab from the Oculus Tools for Unity, but I prefer to use Unity’s native VR Camera for compatibility reasons.

I am aware of “InputTracking.GetLocalPosition(VRNode.RightEye)”, but this gets the “position of node local to its tracking space”, which isn’t what I need. I need the world positions.

Cheers, Wendi

thep3000 · August 26, 2015, 10:09pm

To get world position of the nodes independent of specific camera, transform the positions by the inverse rotations.

Vector3 left = Quaternion.Inverse(InputTracking.GetLocalRotation(VRNode.LeftEye)) * InputTracking.GetLocalPosition(VRNode.LeftEye);

Vector3 right = Quaternion.Inverse(InputTracking.GetLocalRotation(VRNode.RightEye)) * InputTracking.GetLocalPosition(VRNode.RightEye);

If you want these in the coordinate system of a specific camera, you’ll then have to transform by the world to camera matrix:

Matrix4x4 m = Camera.main.cameraToWorldMatrix;
Vector3 leftWorld = m.MultiplyPoint(left);
Vector3 rightWorld = m.MultiplyPoint(right);

Or you can alternatively invert the rotation of the camera and multiply it by the position, but using the cameraToWorld matrix is more efficient.

aurepito · January 7, 2017, 12:33pm

Can you give a screenshot ?

WendelinReich1 · April 3, 2017, 9:37am

thep3000:

To get world position of the nodes independent of specific camera, transform the positions by the inverse rotations.
Vector3 left = Quaternion.Inverse(InputTracking.GetLocalRotation(VRNode.LeftEye)) * InputTracking.GetLocalPosition(VRNode.LeftEye);

Vector3 right = Quaternion.Inverse(InputTracking.GetLocalRotation(VRNode.RightEye)) * InputTracking.GetLocalPosition(VRNode.RightEye);
If you want these in the coordinate system of a specific camera, you’ll then have to transform by the world to camera matrix:
Matrix4x4 m = Camera.main.cameraToWorldMatrix;
Vector3 leftWorld = m.MultiplyPoint(left);
Vector3 rightWorld = m.MultiplyPoint(right);
Or you can alternatively invert the rotation of the camera and multiply it by the position, but using the cameraToWorld matrix is more efficient.

Hi @thep3000 , as of Unity 5.6 on Gear VR, your solution doesn’t return a correct result. I’ve tested this with the following self-contained script (which can be placed anywhere in the scene). It shows a small sphere in front of one eye at a small distance in the direction of Camera.main.transform.forward.

using UnityEngine;
using UnityEngine.VR;

public class EyeTest : MonoBehaviour
{
    public float distanceFromEye = .2f;
    public bool showInFrontOfLeftEye = true;

    Transform smallSphere;
    private Camera cam;

    void Start()
    {
        cam = Camera.main;

        smallSphere = GameObject.CreatePrimitive(PrimitiveType.Sphere).transform;
        smallSphere.localScale = new Vector3(distanceFromEye, distanceFromEye, distanceFromEye) / 10f;
    }

    void Update()
    {
        Vector3 left = Quaternion.Inverse(InputTracking.GetLocalRotation(VRNode.LeftEye)) * InputTracking.GetLocalPosition(VRNode.LeftEye);
        Vector3 right = Quaternion.Inverse(InputTracking.GetLocalRotation(VRNode.RightEye)) * InputTracking.GetLocalPosition(VRNode.RightEye);

        Vector3 leftWorld, rightWorld;

        // neither two solutions are quite correct (Gear VR, Unity 5.6), but the second gives a better result
        if (Time.time < 10f)
        {
            // from https://forum.unity3d.com/threads/world-positions-of-left-right-eye.350076/
            Matrix4x4 m = cam.cameraToWorldMatrix;
            leftWorld = m.MultiplyPoint(left);
            rightWorld = m.MultiplyPoint(right);
        }
        else
        {
            leftWorld = cam.transform.TransformPoint(left);
            rightWorld = cam.transform.TransformPoint(right);
        }

        smallSphere.position = (showInFrontOfLeftEye ? leftWorld : rightWorld) + cam.transform.forward * distanceFromEye;
    }
}

Both in the editor and on the device, the sphere shows up before the camera (on the device, it’s obviously in front of the left eye, as desired). It yaws correctly when I turn the head. But:

it seems clearly closer to the camera on the device, as compared to the editor; and
it changes position when I pitch (look up/down). I’m guessing this means your calculation somehow ignores Unity’s neck model; as this doesn’t happen in the editor.

Could you resolve this?

thep3000 · April 3, 2017, 6:54pm

Right so the step that was missing is that you only want to transform the offset of the eyes by the camera-to-world transform. So this step was missing:

        Vector3 offset = (left - right) * 0.5f;

        Matrix4x4 m = cam.cameraToWorldMatrix;
        leftWorld = m.MultiplyPoint(-offset);
        rightWorld = m.MultiplyPoint(offset);

Here’s a modified version of your sample:

using UnityEngine;
using UnityEngine.VR;
public class EyeTest : MonoBehaviour
{
    public float distanceFromEye = .4f;
    public bool showInFrontOfLeftEye = true;
    Transform smallSphere;
    private Camera cam;
    void Start()
    {
        cam = Camera.main;
        smallSphere = GameObject.CreatePrimitive(PrimitiveType.Sphere).transform;
        smallSphere.localScale = new Vector3(distanceFromEye, distanceFromEye, distanceFromEye) / 10f;
    }
    void Update()
    {
        Vector3 left = Quaternion.Inverse(InputTracking.GetLocalRotation(VRNode.LeftEye)) * InputTracking.GetLocalPosition(VRNode.LeftEye);
        Vector3 right = Quaternion.Inverse(InputTracking.GetLocalRotation(VRNode.RightEye)) * InputTracking.GetLocalPosition(VRNode.RightEye);
        Vector3 leftWorld, rightWorld;
        Vector3 offset = (left - right) * 0.5f;

        Matrix4x4 m = cam.cameraToWorldMatrix;
        leftWorld = m.MultiplyPoint(-offset);
        rightWorld = m.MultiplyPoint(offset);
        smallSphere.position = (showInFrontOfLeftEye ? leftWorld : rightWorld) + cam.transform.forward * distanceFromEye;
    }
}

And resulting editor screenshot attached (tested with OpenVR).

Note that the sphere is not in the center on OpenVR because of its asymmetric off-center projection. Also note that I pushed out the distanceFromEye to .4f to be further out than the default camera near plane.

I didn’t try this with GearVR but I expect it will work all-the-same.

WendelinReich1 · April 4, 2017, 8:42am

Thanks @thep3000 , it works as you say on GearVR & Daydream. I think you should include your solution in the documentation for InputTracking.GetLocalPosition(), it’s a common enough problem!

Actually, come to think of it, I think it should be an API call. Why not add an InputTracking.GetWorldPosition()?

thep3000 · April 4, 2017, 3:45pm

I’ll pass your feedback for a world accessor on to the team that is working on input now.

And in the meantime, I’ll add a version of this to the current docs.

Thanks!

Cleverlie · May 31, 2017, 1:46pm

Hi people, I have a question regarding native cardboard VR in unity 5.6, it is related to this thread so I’ll write here, maybe some of the devs who already answered above can illuminate me:

I have a setup of cameras like this:

where:
A - empty game object which is the world offset (or head position in the game), this is freely moved by scripts

B - camera with target eye = left and layermask to show only left-eye stuff
C - camera with target eye = right and layermask to show only right-eye stuff

D - center camera with target eye = none, which is used when I compile to WebGL so instead of VR I have only one camera rendering full screen, also I use this camera to remote cast to chromecast.

so my setup has flexibility to render both eyes individually with exclusive content for each eye (like different 360 skyboxes) instead of having one single camera with target eye = both, also it has flexibility to cast the “center camera” to chromecast while the smartphone still has the eyes rendering to the phone screen.

my problem now is that I did a quick test because it seems that when you don’t use the “both” option in a single camera, each different camera for left and right eyes, seems to be only receiving the local rotation of VR, and not also the local position, what I mean is this:

in a standard setup you have a single camera with “both” option set for target eye, that will render two different invisible cameras and their positions will be offseted a couple of centimeters depending on “stereo separation” parameter, also I suppose that this two runtime-created cameras has a rotation that is influenced by “stereo convergence” parameter.

the problem is that when I have my custom cameras with each eye set for each camera, the “stereo separation” doesn’t do anything to their local positions, aka offset from head center, and I have to do the offset manually like in old cardboard versions without native integrations, where you have an empty “head” game object which is the one rotated by gyroscope values, and two fixed cameras that are offseted and child of this object, the cameras don’t rotate locally, they inherit rotation from the parent.

with my setup I end up with things like “wrong behaviour”, and what I want is “correct behaviour”:

I think I could do something like this if I have full control of the cameras rotation and position via InputTracking.GetLocalPosition(VRNode.RightEye) etc. but in that case I would be overriding the unity’s native black magic positioning, which I don’t know at which moment in script execution happens, in update? in fixed update? in late update? how can I be sure that I can override the positioning and rotation, while maintaining the native integration of the rendering etc?

Cleverlie · May 31, 2017, 1:53pm

also, adding to my question above, it would be great if the docs explain explicitly what happens when VR mode is active, because is really hard to debug this things in the phone, native integration with VR is great but having it as a black box without knowin exactly what unity is doing to your cameras is bad, you can’t really know if the cameras are being only locally rotated, or also offseted by eyes separation, or if it happens in local o world coordinates, or if it happens in “both” or “left/right” or “none” modes, you have to figure out everything by yourself doing complicated stuff, or having a really good eye to see that the projections are all wrong…

Selzier · May 31, 2017, 2:22pm

I don’t have any answers for you Cleverlie, but I will say that all the “Left/Right” world camera got really annoying to me, so we made a shader that renders to the left or right eye specifically, using a single camera only, without using layers/masks.

That being said, I haven’t tested anything like using the middle camera for Chomecast, so that obviously is something beyond what my shader does. But you may be able to write a shader to achieve what you’re trying to do.

greggtwep16 · May 31, 2017, 3:51pm

I would think the script posted above with the sphere in front of the eye would work if you modify it to update the position for your left and right cameras. Offset would be your separation you want.

I definitely second your notion on both documentation on what Unity is doing to the camera and when in VR. Mobile VR even for a camera set to “both” has a habit of essentially locking you out of the transform. I assume this is either done on purpose to lock you out and prevent nausea or because the native integration actually happens later than lateUpdate(). This will be different behavior than in editor because the camera will be updated at different times. In either case transparency on what is happening would be appreciated.

@thep3000 @joejo Do the updates happen after lateUpdate() when mobile VR is using native integration? Is there any documentation on the flow of VR camera’s or a way to modify the transform after the update from the HMD?

DavidMiranda · June 8, 2017, 1:29pm

Im having a huge pain trying to replace:
Vector3 eyePos = SceneCamera.transform.TransformPoint(SteamVR.instance.eyes[1].pos);

With built-in stuff. I have tried the content shown here with no luck. Any tips?
I need to get the world position for both eyes but I don’t want to use valve stuff. That line is even giving me incorrect position, as you can see here
Thanks.

Matt_D_work · June 13, 2017, 5:30pm

You are correct, in implicit camera update modes, the position/rotation is set after lateUpdate monobehaviour update step.
The reason we go as late as possible is to minimize latency between position/rotation sampling and update and the eventual render step. The lower the latency here, the better the overall feel for the user.

greggtwep16 · June 13, 2017, 6:14pm

Thanks for the response!

The effect that this creates is the feeling of “transform lockout” on the camera. If you want to move the position then you must create a parent object to do so, regardless if it would have made sense. I don’t recall this happening on desktop vr when teleporting, I believe I could just update the camera position, is it done differently on desktop VR?

I certainly understand the need to do it as late as possible wouldn’t it still be possible to do this without the effect of locking the developer out of the transform? Essentially the engine always factoring in both what the user has specified and what the HMD is telling it. So for position it would be the transform position + VR SDK position. Similarly for rotation.

If this isn’t desired to be done, perhaps it can be mentioned more prominently in the documentation. I get a decent amount of support questions in my asset and have to explain that the way Unity currently works that any updates to the camera transform will be ignored and to create a parent.

plmx · July 11, 2017, 9:48am

@greggtwep16 @Matt_D_work I posted about the issue of not being able to control the camera - vs. being able to control the controllers - here (On native Unity VR support: Camera vs. Controllers - Unity Engine - Unity Discussions). I suggested that there be some way of controlling the camera. Would be great to have that.

@thep3000 Thanks for the code for the eye tracking. I run into the following issue however: When tracking the eyes like this and moving the head sideways into a wall, the user is able to look through the wall before the above code indicates that the user is inside the wall. What might be the cause of this? Is this a clipping issue?

ShawnFeatherly · February 13, 2018, 10:53pm

I got this working with:

Vector3 headPosition = Camera.main.transform.position;
Vector3 handOrigin = UnityEngine.VR.InputTracking.GetLocalPosition(UnityEngine.VR.VRNode.RightHand);
handOrigin += headPosition;

This seems shorter that the code above. Is there some reason it’d be faulty? It seems like both rely on Camera.main. It seems sketchy that the headPosition with the original code (also below) doesn’t match with Camera.main.transform.position

Vector3 headPosition = Quaternion.Inverse(UnityEngine.VR.InputTracking.GetLocalRotation(UnityEngine.VR.VRNode.Head)) * UnityEngine.VR.InputTracking.GetLocalPosition(UnityEngine.VR.VRNode.Head);
headPosition = Camera.main.cameraToWorldMatrix.MultiplyPoint(headPosition);

zezba9000 · March 23, 2019, 9:02pm

InputTracking.GetLocalPosition(XRNode.LeftEye) just returns a zero vector in a “Mock HMD”.
Is this a bug in Unity 2018.3.9 ?

Matt_D_work · March 24, 2019, 8:45pm

i would say yes. please raise a bug on that

EricHartman · March 26, 2019, 6:56pm

Thank you, have problems with it. now it done.

nasos_333 · April 28, 2020, 3:25pm

Hi,

What is the latest way to get the left - right camera position in onRender for an image effect ?

I have spent an enormous ammount of hours with no luck, tried all suggestions in
https://discussions.unity.com/t/747284

Thanks

Using unity 2019.3.7f1 and MockHMD

Topic		Replies	Views
Gear VR left and right eye world position Unity Engine XR	1	1431	November 12, 2016
Finding VR Eye Position (including lens distance offset) - holographic sight Unity Engine XR	1	3057	August 9, 2018
[fixed] Wanted: Rift camera reset function Unity Engine 5-1-beta	24	21573	May 13, 2015
5.4.0 b9 Get VR eye position and rotation? Unity Engine 5-4-beta	2	4766	March 9, 2016
_WorldSpaceCameraPos does not work correctly for VR stereo camera in single pass renderying Unity Engine XR , URP , Question , com_unity_render-pipelines_universal	4	1141	March 1, 2024

World positions of left/right eye

Related topics