`EyeGaze:/EyeGaze` device not added to system

[anon20000101]

I’m trying to setup gaze interactor with quest pro connected via link cable (windows standalone platform). I’m following this guide in the documentation. Both the OculusEyeGazeInputAdapter are logging that eye tracking is successfully started for each eye. However, on Update, they’re giving the following error:

InvalidOperationException: Cannot queue state event for control ‘Pose:/EyeGaze/pose’ on device ‘EyeGaze:/EyeGaze’ because device has not been added to system
UnityEngine.InputSystem.InputSystem.QueueDeltaStateEvent[TDelta] (UnityEngine.InputSystem.InputControl control, TDelta delta, System.Double time) (at ./Library/PackageCache/com.unity.inputsystem@1.7.0/InputSystem/InputSystem.cs:2687)
UnityEngine.XR.Interaction.Toolkit.Samples.MetaGazeAdapter.OculusEyeGazeInputAdapter.Update () (at Assets/Samples/XR Interaction Toolkit/2.5.2/Meta Gaze Adapter/Runtime/Scripts/OculusEyeGazeInputAdapter.cs:143)

It seems like I need openxr plugin enabled to be creating these devices, but I’m not allowed to have both the oculus plugin enabled and the openxr plugin enabled at the same time.

Unity version: 2022.3.11f1

Package Versions:
com.unity.xr.interaction.toolkit@2.5.2
com.meta.xr.sdk.utilities@57.0.0-preview
com.unity.xr.oculus@4.1.1
com.unity.xr.openxr@1.8.2

[ericprovencher]

So unfortunately the way this script is setup, it’s a little confusing because it doesn’t really support this mode.

The last time I tested, I got it working only in the editor, by having the openxr package installed, but using the Oculus xr loader back end, because the eye device didn’t seem to be detected with the OpenXR package active, but the OpenXR package houses some of the input device scripts used in the eye gaze XRI sample. In builds this workflow didn’t really work.

We haven’t done extensive testing with the latest Meta integration packages, and the Meta XR Feature group in your screenshot, which is added by those packages, is known to have some conflicts with some of the openxr features that Unity supports out of the box.

Unfortunately getting everything to play nicely together is a challenge, but it’s something we’re actively working on.

My advice to you if eye gaze functionality is important to you, is to use the Eye Gaze sample as a learning material to register the input device that hold the eye gaze transform, and then refer to meta’s documentation on supporting the eye gaze vector.

In XRI, the eye gaze functionality uses the industry standard Eye gaze openxr extension, which works with HoloLens 2 and other devices, but Meta Quest Pro exposes eye gaze with their own vendor specific spec that isn’t easily exposed to XRI, hence why this adapter sample exists.

[anon20000101]

Doing some digging in the scripts, I don’t understand why I’m getting the error. The adapter script says it’s successfully adding the device to the input system, and I see the EyeGaze and EyeGaze1 devices (for left and right eyes), but I’m still getting the error.

Why is this flow not ideal in a build? I can see how it is pretty hacky, trying to support EyeGaze by using the input system definitions from a package you don’t actually need. Would it be better to create a custom EyeGaze device?

UPDATE:
I figured out what was causing the error. I had two adapter scripts, one for each eye. However, there should only be a single EyeGaze device. The other device becomes invalidated for some reason when both scripts are active. Digging through this, I learned a little more how this flow is supposed to work. I’m going to modify the adapter script so that it fixes some of these limitations (combining both eye inputs into a single one, maybe a custom device instead of relying on OpenXR package)

[ericprovencher]

Feel free to share your changes back with us so we can include them in the future samples.
Indeed removing the OpenXR dependency there is probably for the best, at least until the OpenXR workflow starts to improve on the meta integration asset.

Glad you’re making progress!

[anon20000101]

I’ve got it working! Here are the modifications.

1. Create a custom device.

using UnityEngine.InputSystem;
using UnityEngine.InputSystem.Layouts;
using UnityEngine.InputSystem.XR;
using UnityEngine.Scripting;

namespace UnityEngine.XR.Interaction.Toolkit
{
    [Preserve, InputControlLayout(displayName = "Eye Gaze (Meta)")]
    public class MetaEyeGazeDevice : InputSystem.InputDevice
    {
        [Preserve, InputControl(offset = 0, usages = new[] { "Device", "gaze" })]
        public PoseControl pose { get; private set; }

        /// <inheritdoc />
        protected override void FinishSetup()
        {
            base.FinishSetup();

            pose = GetChildControl<PoseControl>(nameof(pose));
        }
    }
}

2. Modify adapter script to combine both eye poses.

bool AreUnderConfidenceThreshold(OVRPlugin.EyeGazeState left, OVRPlugin.EyeGazeState right)
{
    return left.Confidence < m_ConfidenceThreshold || right.Confidence < m_ConfidenceThreshold;
}

void ProcessPoseInput()
{
    if (!m_OculusEyeTrackingStarted)
        return;

    if (!OVRPlugin.GetEyeGazesState(OVRPlugin.Step.Render, -1, ref m_OculusEyeGazesState))
        return;

    var leftEyeGazeState = m_OculusEyeGazesState.EyeGazes[(int)Eye.Left];
    var rightEyeGazeState = m_OculusEyeGazesState.EyeGazes[(int)Eye.Right];

    if (m_CheckConfidence && AreUnderConfidenceThreshold(leftEyeGazeState, rightEyeGazeState))
        return;

    var combinedIsValid = leftEyeGazeState.IsValid && rightEyeGazeState.IsValid;

    var leftEyePose = leftEyeGazeState.Pose.ToOVRPose();
    var rightEyePose = rightEyeGazeState.Pose.ToOVRPose();

    var combinedPosition = Vector3.Lerp(leftEyePose.position, rightEyePose.position, 0.5f);
    var combinedOrientation = Quaternion.Slerp(leftEyePose.orientation, rightEyePose.orientation, 0.5f);

    m_PoseState.isTracked = combinedIsValid;
    m_PoseState.trackingState = combinedIsValid ? InputTrackingState.Position | InputTrackingState.Rotation : InputTrackingState.None;
    m_PoseState.position = combinedPosition;
    m_PoseState.rotation = combinedOrientation;
}

3. Use the new eye gaze device instead of the OpenXR one.

/// <summary>
/// See <see cref="MonoBehaviour"/>.
/// </summary>
protected void Awake()
{
    InputSystem.InputSystem.RegisterLayout(typeof(MetaEyeGazeDevice),
        "EyeGaze",
        matches: new InputDeviceMatcher()
            .WithInterface(XRUtilities.InterfaceMatchAnyVersion)
            .WithProduct("Meta Eye Gaze Adapter"));
}

void AddEyeGazeDevice()
{
    if (m_EyeGazeDevice != null && m_EyeGazeDevice.added)
        return;

    m_EyeGazeDevice = InputSystem.InputSystem.AddDevice<MetaEyeGazeDevice>();
    if (m_EyeGazeDevice == null)
    {
        Debug.LogError("Failed to create Eye Gaze device.", this);
        m_PoseControl = null;
        return;
    }

    m_PoseControl = m_EyeGazeDevice["pose"];
}

Lastly, you can remove the references to OpenXR.

[ericprovencher]

Very nice thanks a lot for sharing!

[jleeTRIPP]

As a further note, controller/hand-tracking does not work when running on device on Quest* (Pro or 3) when configured per documentation.

When OculusXR plugin is installed(active via XR Plug-in Management), and OpenXR package is installed (NOT active via XR Plug-in Management), controller/hand-tracking no longer works when running on device on Quest* (Pro or 3). Controller/hand-tracking otherwise works fine in-editor.

I can Oculus Button and then my controllers are working. “Resume” app and Controllers display for a second, and disappear like they were disconnected. This is a huge blocker with non-functional controller/hand-tracking to trade for eye-tracking.

I’ve been able to debug this down to it only happens when the OpenXR package is installed with the Oculus Plugin – per the configuration detailed in the Meta Gaze adapter documentation. Input Debugger says controllers are connected and generating input. Dumping connection state from InputSystem says both controller/hands devices are connected and available. It’s just A/B difference with/without OpenXR package installed.

My current workaround is to use jasonboukheir3’s patches (with some mods), and so the eye-tracking works without OpenXR installed at all. The code would still fall-back to OpenXR when available, but not when built with the OculusXR plugin.

Configuration details:

• Unity 2021.3.32f1
• XRIT 2.5.2
• Oculus Integration v57
• OpenXR 1.7.0 (or 1.8.2, 1.9.1)
• USE_INPUT_SYSTEM_POSE_CONTROL
• eye-tracking enabled in OculusProjectConfig, and in the AndroidManifest.xml

[BladestormX]

I’ve got it working! Here are the modifications.

1. Create a custom device.

using UnityEngine.InputSystem;
using UnityEngine.InputSystem.Layouts;
using UnityEngine.InputSystem.XR;
using UnityEngine.Scripting;

namespace UnityEngine.XR.Interaction.Toolkit
{
    [Preserve, InputControlLayout(displayName = "Eye Gaze (Meta)")]
    public class MetaEyeGazeDevice : InputSystem.InputDevice
    {
        [Preserve, InputControl(offset = 0, usages = new[] { "Device", "gaze" })]
        public PoseControl pose { get; private set; }

        /// <inheritdoc />
        protected override void FinishSetup()
        {
            base.FinishSetup();

            pose = GetChildControl<PoseControl>(nameof(pose));
        }
    }
}

2. Modify adapter script to combine both eye poses.

bool AreUnderConfidenceThreshold(OVRPlugin.EyeGazeState left, OVRPlugin.EyeGazeState right)
{
    return left.Confidence < m_ConfidenceThreshold || right.Confidence < m_ConfidenceThreshold;
}

void ProcessPoseInput()
{
    if (!m_OculusEyeTrackingStarted)
        return;

    if (!OVRPlugin.GetEyeGazesState(OVRPlugin.Step.Render, -1, ref m_OculusEyeGazesState))
        return;

    var leftEyeGazeState = m_OculusEyeGazesState.EyeGazes[(int)Eye.Left];
    var rightEyeGazeState = m_OculusEyeGazesState.EyeGazes[(int)Eye.Right];

    if (m_CheckConfidence && AreUnderConfidenceThreshold(leftEyeGazeState, rightEyeGazeState))
        return;

    var combinedIsValid = leftEyeGazeState.IsValid && rightEyeGazeState.IsValid;

    var leftEyePose = leftEyeGazeState.Pose.ToOVRPose();
    var rightEyePose = rightEyeGazeState.Pose.ToOVRPose();

    var combinedPosition = Vector3.Lerp(leftEyePose.position, rightEyePose.position, 0.5f);
    var combinedOrientation = Quaternion.Slerp(leftEyePose.orientation, rightEyePose.orientation, 0.5f);

    m_PoseState.isTracked = combinedIsValid;
    m_PoseState.trackingState = combinedIsValid ? InputTrackingState.Position | InputTrackingState.Rotation : InputTrackingState.None;
    m_PoseState.position = combinedPosition;
    m_PoseState.rotation = combinedOrientation;
}

3. Use the new eye gaze device instead of the OpenXR one.

/// <summary>
/// See <see cref="MonoBehaviour"/>.
/// </summary>
protected void Awake()
{
    InputSystem.InputSystem.RegisterLayout(typeof(MetaEyeGazeDevice),
        "EyeGaze",
        matches: new InputDeviceMatcher()
            .WithInterface(XRUtilities.InterfaceMatchAnyVersion)
            .WithProduct("Meta Eye Gaze Adapter"));
}

void AddEyeGazeDevice()
{
    if (m_EyeGazeDevice != null && m_EyeGazeDevice.added)
        return;

    m_EyeGazeDevice = InputSystem.InputSystem.AddDevice<MetaEyeGazeDevice>();
    if (m_EyeGazeDevice == null)
    {
        Debug.LogError("Failed to create Eye Gaze device.", this);
        m_PoseControl = null;
        return;
    }

    m_PoseControl = m_EyeGazeDevice["pose"];
}

Lastly, you can remove the references to OpenXR.

When I try this solution, I get this error:

Any idea what might be wrong?

[naskosn]

@BladestormX

When I try this solution, I get this error:

Any idea what might be wrong?

Did you manage to solve the problem?

[Wisteria5]

@BladestormX

When I try this solution, I get this error:

Any idea what might be wrong?

Simplest fix for this for me, was to put the MetaEyeGazeDevic.cs file into the same directory as the OculusEyeGazeInputAdapter.cs file.