Gaze selection possible in fully immersive apps?

We have a fully immersive app using the built-in render pipeline. We would like to emulate the gaze selection used in the OS in our menus, but looking over the docs, it seems PolySpatial is required for this. I just wanted to confirm here, is that correct? Is there any pathway on the roadmap for this to be possible in apps that do their own stylized rendering (and thus can’t use PolySpatial)?

thanks,

Mike

1 Like

I would like to know this also.

@puddle_mike I’m not sure this has anything to do with anything, but the gaze and select works in Apple’s “Hello World” outer space fully immersive demo.


I just rewatched Apple’s video on Unity for Fully Immersive and it looks like it should work eventually.

I am also stuck here. Since there is no controller, I am a bit confused about the best practices for selecting UI elements such as buttons without the gaze and tap gesture. Any light on this would be super appreciated!

2 Likes

And then going forward, if we’re trying to test stuff in Fully Immersive, should we be using Unity’s VR template in Unity Hub (then adding VisionOS packages) or the VisionOS Template from the google drive file?

Hey there!

This was an oversight on our part. Gaze input was broken/removed from our com.unity.xr.visionos package in an earlier update (prior to 0.1), and never restored. It should be back in the next version we ship. I’ll update this thread when it is available.

should we be using Unity’s VR template in Unity Hub (then adding VisionOS packages) or the VisionOS Template from the google drive file?

You should use the visionOS template. We will try to make sure it is easy to switch this to VR mode.

But to be clear, Gaze input is head gaze, not eye gaze, correct? My understanding is that Apple does not expose the eye data directly, only highlight states on special objects you define.
My main question is if it’s possible to build an API such that fully immersive apps can define objects they’d like to know when the eyes are locked onto. If not, I can bring this up with Apple…
And similarly, is the OS-level pinch gesture exposed to fully immersive apps? I have created my own for now, but I’m sure its worse that the OS implementation :slight_smile:

1 Like

Here is Apple’s documentation about the API we’re using. It doesn’t explicitly say anything eye gaze, but when I tested this the ray points where my eyes are looking, not just where my head was facing. I tested this by doing a Physics.Raycast using the origin/direction from the spatial event and placing a sphere where it hit.

This API returns the same SpatialEventCollection.Event struct that we get in Mixed Reality which includes a selectionRay property. Based on your device settings, this can be driven by eye gaze, wrist direction, or a variety of input sources. By default it is driven by eye gaze.

That is correct, except for that first event of a pinch gesture, both in Mixed Reality and Virtual Reality. You get one data point when the user initiates an interaction, but you can’t arbitrarily request/poll for eye gaze like you can for head pose.

This is unfortunately not possible at the moment. The user has to initiate the interaction with a pinch. So in practice, users can click buttons in Virtual Reality with gaze/pinch, but we can’t implement a hover state to show them what button will be clicked ahead of time. Please do bring this feedback to Apple. This issue has come up before and it helps to amplify the signal with more voices/perspectives.

Yeah that’s what we’re talking about exposing here. As the Apple docs say, you can’t rely on interacting with OS-level objects or UI in Virtual Reality, but they send an event whenever a pinch is detected.

Thanks for the clarification. We’ll bring this up to Apple in the hopes that this unblocks the ability to do highlight states in our custom game UI’s (for fully immersive apps).

Do you have an example of how to handle the existing pinch event on the Unity side? (documentation, code sample, etc) Note that we do not have PolySpatial stuff in our project…

Unfortunately we haven’t shipped an update yet to use the pinch event in Unity. When it is ready, it will be similar to the SpatialPointerDevice you can use with PolySpatial in mixed reality. I gave a little more info in this thread.

PSA for other devs: I think for now, devs in full VR mode (i.e using the Metal rendering path) won’t have any options for utilizing the platform level gaze + pinch. For privacy reasons, Apple will only expose the gaze direction to devs when your fingers pinch (an implicit “permission”). Maybe some day they will let devs ask permission of users to expand the possibilities…
That leaves the only potential option to switch between Shared Space/Mixed Really and full VR so you can handle your UI in the Shared Space using gaze control. But that, at the moment, isn’t supported by Unity:
https://discussions.unity.com/t/switching-between-mixed-and-virtual-reality-while-playing/294323/8

We are hitting this limitation at the moment as well. Ideally we would love to have both, permissions to be able to poll eye tracking data for rays and the ability to switch from VR to AR/MR mode.

As a side ntoe, the switch to AR/MR mode is basically something we already do in mobile devices with AR foundation. Why isnt that a possibility here? (Even if it would still limit functionality, it would allow us to use existing content)

Isn’t this already available without PolySpatial as shown in the VR samples included in the Apple visionOS XR Plugin package?

Is it supposed to work now? I made a simple replication project and when I switch it to MR mode it gives me gaze/pinch events, but when I switch it to VR mode it does not.

image
Sorry, I couldn’t resist :laughing:

But seriously, we are working toward this. As you say, on mobile devices it’s as easy as just turning AR on and off, which will turn on the camera and start running SLAM, pretty much whenever you want.

On visionOS, however, we aren’t always in full control of rendering like we are on mobile devices. The app setup for Mixed Reality with PolySpatial is currently a little different from the one we use for Virtual Reality because of this. For Virtual Reality, we need to carefully control the sequence of events where we set up the LayerRenderer for VR rendering, and because of the way SwiftUI is set up, the swift code for your app needs to have all of your volumes, immersive spaces, etc. implemented ahead-of-time, so the choice to include an immersive space for compositor services rendering must always be made at compile time.

Suffice it to say, while there are challenges to getting this working, it is certainly possible, and we’re doing our best to get it to you as soon as we can.

1 Like

See my cheeky meme above for whether or not mode switching works now… it does not. :wink:

Edit: nevermind, I think I see what you’re asking now… Does gaze/pinch input work in VR? Yes!

But I think what you’re saying in the next sentence is that when switching your app mode to VR in project settings, input stops working. That is because, as with rendering, pinch/gaze input has differences between VR and MR, and thus you will need to change how you implement input depending on the app mode.

In MR, we get a targeted entity for a given pinch gesture, which is especially useful in bounded mode, when you don’t have access to the gaze vector. Without this, you would need to do a sphere cast around the interaction position to find what object was being interacted with, and in some cases that would give you the wrong result.

In VR, however, there are no RealityKit entities, so this isn’t an option. Thankfully, because you’re always in an immersive space, you always get a gaze vector on the first frame of the pinch, which means you can do a regular old raycast in Unity. Furthermore, the machinery for finding the GameObject backing a given RealityKit entity all lives in the PolySpatial package, so VR input needs to function independently.

This is why we have a bit of a fragmented input story for visionOS. For Mixed Reality, you need to use SpatialPointerDevice, which provides the data about pinch gestures, including the targeted entity. For Virtual Reality, you need to use VisionOSSpatialPointerDevice, which provides the same pinch gesture data, excluding targeted entity. As @dariony points out, the VR samples in com.unity.xr.visionos (Apple visionOS XR Plugin package) show you how to use VisionOSSpatialPointerDevice with XRI, Unity UI, and regular C# code. When pivoting to VR, you’ll need to update your input setup, but it’s not that different, and the samples should be your guide.

Thanks for reaching out, and good luck!

2 Likes