Mixed Reality Interactions not working when I switch over to Virtual Reality Mode

I’m using the following code to with the gaze and pinch interaction on floating spheres to trigger some behavior and it works fine in the Mixed Reality Mode, but once I switch to Virtual Reality, the same interaction stops working. What should I do?

Thanks so much…

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.InputSystem.EnhancedTouch;
using TouchPhase = UnityEngine.InputSystem.TouchPhase;
using Touch = UnityEngine.InputSystem.EnhancedTouch.Touch;

public class SphereTouchDetector : MonoBehaviour
{
    [SerializeField]
    private InputActionReference m_TouchInputAction;

    private void OnEnable()
    {
        m_TouchInputAction.action.performed += OnTouchPerformed;
        m_TouchInputAction.action.Enable();
    }

    private void OnDisable()
    {
        m_TouchInputAction.action.performed -= OnTouchPerformed;
        m_TouchInputAction.action.Disable();
    }

    private void OnTouchPerformed(InputAction.CallbackContext context)
    {
        if (Touch.activeTouches.Count > 0)
        {
            foreach (var touch in Touch.activeTouches)
            {
                if (touch.phase == TouchPhase.Began)
                {
                    CheckTouch(touch);
                }
            }
        }
    }

    private void Update()
    {
        if (Touch.activeTouches.Count > 0)
        {
            foreach (var touch in Touch.activeTouches)
            {
                if (touch.phase == TouchPhase.Began)
                {
                    CheckTouch(touch);
                }
            }
        }
    }

    private void CheckTouch(Touch touch)
    {
        // Convert touch position to ray
        Ray ray = Camera.main.ScreenPointToRay(touch.screenPosition);
        RaycastHit hit;
        // Perform raycast to check if a sphere was touched
        if (Physics.Raycast(ray, out hit))
        {
            if (hit.collider.gameObject == gameObject)
            {
                // Sphere was touched
                Debug.Log("Sphere Touched: " + gameObject.name);
                GameManager.Instance.SphereClicked(gameObject);
                FreezeSphere();
            }
        }
    }

    private void FreezeSphere()
    {
        Rigidbody rb = gameObject.GetComponent<Rigidbody>();
        if (rb != null)
        {
            rb.velocity = Vector3.zero; // Stop movement
            rb.angularVelocity = Vector3.zero; // Stop rotation
            rb.isKinematic = true; // Make Rigidbody not be affected by physics anymore
        }
    }
}```

Hi there! Thanks for reaching out and sorry to hear you’re having trouble.

This behavior is expected. VR and MR use different (similarly named) input devices. The VR device is called VisionOSSpatialPointerDevice, and the MR device is called SpatialPointerDevice. I know this is not ideal, but there are good reasons why we have it set up this way. Primarily, the MR device requires you to interact with a specific RealityKit object, and gives you an ID for that object when you interact. This functionality requires PolySpatial. The VR device also lives in com.unity.xr.visionos, while the MR device lives in com.unity.polyspatial, which is not required for VR apps. You will need to set up your actions (and/or or scripts) to use both input methods, or use a different scene/configuration for VR builds and MR builds.

In terms of how you have set up this script, you will need to use individual bindings for spatialPointer0 and spatialPointer1, as the VR device does not emulate a touch screen like the MR device does. Take a look at the package samples in com.unity.xr.visionos for an example of how to set up input for VR.

Thank you for the quick response. @mtschoen

I’m starting from scratch using the package sample (Vision OS VR URP). Let’s say I want something to happen when I gaze and pinch on this blue cube. How would I go about making that happen?

Unlike in the MR mode, you don’t actually need to gaze at an object for input to work. Check out the InputTester script for an example of how to react to gaze/pinch input. The blue cubes are set up to use the XR Interaction Toolkit, so input handling is abstracted.

Thanks once again. I was able make it work. Although now I cannot seem to remove the “icy structure and green panel shapes” that come from the VR Sample URP.

I was able to remove the UI elements and the blue cubes, but not the “ice room”. It only appears when I run the game on the device though, not in the editor or the player.
@mtschoen

I think you’re referring to the ARKit meshing? It uses a transparent blue material that looks like ice, I guess :laughing:

This is all set up underneath the XR Origin. In the screenshot below, the Mesh Manager object is responsible for the blue meshes, and the green planes are created by the ARPlaneManager component on the XR Origin.

1 Like

Hello~ Is ther any reason why VR mode doesn’t emulate a touch screen while MR does?

Thanks!!

Not particularly, no. It’s something that didn’t quite work out as well as we had hoped for MR, given the reliance on a camera to come up with a fake “screen space” position. We can’t be sure that the main camera is always appropriate for this, and sometimes you get unexpected results when trying to use this touch screen for UI input, etc. Furthermore, as the user moves their fingers, we have an updated world space “interaction position” in MR that we don’t have in VR. Instead, for VR we have the interactionRayRotation, which is already a “fake” control given that we compute it using the inputDevicePosition as it changes over time. Adding another layer of “fake” touchscreen data seemed like it would only confuse users who expect it to work exactly like existing touch screen support for phones and tablets.

If this is something you would still like to see us try and add as a feature, please submit it as an idea on the visionOS roadmap.

Hi @mtschoen, thank you for your kind response!
I’m new to XR development and I’m a little confused about the actual physical meaning of interactionRayRotation. How is that different from the gaze ray? There’s no doc on this and it would be of great help if you could explain this in a little bit more detail.
According to the changelog, interactionRayRotation was first introduced in com.unity.xr.visionos 1.0.3 to handle slider inputs. From the source code (I’m on 1.1.6 now), it is calculated as follows:

// Compute interaction ray rotation based on device position
var rayOrigin = inputEvent.rayOrigin;
var rayDirection = inputEvent.rayDirection;
if (m_StartInputDevicePositions.TryGetValue(interactionId, out var startInputDevicePosition))
{
    // Calculate start position at arbitrary distance, roughly within arms' reach
    var gazeRay = new Ray(rayOrigin, rayDirection);
    var startPosition = gazeRay.GetPoint(0.5f);

    // Update current position based on distance inputDevicePosition has moved
    var currentPosition = startPosition + (inputDevicePosition - startInputDevicePosition);
    var interactionRayDirection = Vector3.Normalize(currentPosition - rayOrigin);
    state.interactionRayRotation = Quaternion.LookRotation(interactionRayDirection);
}

Let’s say that the we (the camera) are at the origin and we have a horizontal slider right in front of us. We want to move the slider to the right. Here’s how we do: we pinch our fingers at a random position, hold the “pinch” and move our hand horizontally to the right for a small distance (e.g. 10cm), and then release the pinch gesture. During the whole time, our eyes follow the slider to the right while our head keeps still.

In this case, from my understanding, we have rayOrigin=0 all the time, startPosition 50cm along the gaze ray, and inputDevicePosition-startInputDevicePosition is 10cm horizontally to the right. We simply look at interactionRayDirection (since it means the same as interactionRayDirection), and for direction we can ignore the normalization step, so after moving and before releasing the pinch gesture, we have
interactionRayDirection = startPosition + hand offset - rayOrigin = (50cm along the gaze ray) + (10cm to the right)
It’s a direction after the gaze direction “drifts” to the right. What does this direction give us? What is the origin of the interaction ray? Since our hand has stopped moving, the slider shouldn’t have any “intention” to move.
Please feel free to point out where I’m wrong. The question is a bit long and I’m trying my best to explain my thoughts clearly. Thank you again for your time and patience. Looking forward to your reply :slight_smile:

Also in the official URP sample included in com.unity.xr.visionos, we have a Ray Origin gameobject with a TrackedPoseDriver component, whose Rotation Input is binded to interactionRayRotation. If I change this binding to startRayRotation, I’m still able to move the cube interactables, but not the slider. This also confuses me.

You don’t need to follow the slider with your eyes. interactionRayRotation uses the gaze ray as a starting point, and then deflects that ray based on how the pinch moves. You can sort of think of it like you have a string tied around the ray and you’re tugging it this way and that. You should see one of the red “laser beam” objects in the scene move around as you pinch and move your hand. Simply put, if you gaze at an object, pinch and drag your fingers to the right, the interaction ray will start out matching the gaze ray, and rotate clockwise on the Y axis around the ray origin at your eyes.

I’m confused by this question. If your hand is not moving, interactionRayRotation also does not move. It will only rotate the initial gaze ray if your hand moves from the original location where you pinched. Maybe it’s easier to think of it like a joystick? Like in the game Starfox, you start out with the reticle in the center of the screen, and then when you push the stick to the right, the reticle moves to the right. The trajectory that your ship shoots along is a ray from the ship to the reticule, and it “rotates” around as you push the reticule around the screen with the stick. The same thing is happening here. The gaze ray gives us a starting point (like starting with the reticle at the center) and as you move your hand, it’s like the joystick pushing the reticle around.

This is expected. startRayRotation does not change throughout the interaction, and always represents the ray provided by the OS to represent the user’s gaze. You can move the interactables because their motion is based solely on your hand movement. The gaze ray is only used to figure out which object you are interacting with. For a UI slider, we need to use raycasts because that’s how the UI system is set up, hence the need for interactionRayRotation.

Hopefully that clears some of this up?

OHHHH this really helped me out. I thought that StartRayOrigin and StartRayDirection would also update realtime throughout the interaction, and that’s where things start to get confusing. I did some experiments and found out that these two only update at the beginning of each interaction.
This really should be added to the document. Might be a great time saver for lots of noobs (like me :laughing:).

Thank you again for your patience and kind response!! You are so inspiring :grin: