How to handle vision OS sight direction from code?

Target Platforms XR
, ,
1

I’m not clearly understand how to handle visionOS gaze direction. Is there some code samples or something, where I can see how to write own handling for view direction.

2

Hi there!

For privacy reasons, Apple does not expose the gaze vector to application code. The one exception is for the gaze/pinch interaction. The first event fired by the system includes the gaze vector at the moment the user pinches their fingers which was used to raycast into the RealityKit scene. We expose that through the SpatialPointerDevice via the startInteractionRayOrigin and startInteractionRayDirection. These Vector3 controls will give you the ability to cast a ray in Unity to do your own intersection test.
We recommend that, for mixed reality apps. you use the XR Interaction Toolkit and XRTouchSpaceInteractor provided by the com.unity.polyspatial.xr package. This will take advantage of the targetId control, which identifies which specific RealityKit entity the user interacted with. The targetID control maps to a GameObject instanceID on the Unity side, which we can use to look up the specific GameObject and Interactable component without doing our own raycast.

Also please note that this ray origin/direction is not provided in the simulator, and I think it is only provided on the device if you have an immersive space open (meaning an unbounded volume camera on the Unity side). Furthermore, there is a bug in the OS at the moment that reports this information incorrectly, so you won’t be able to rely on this data until that is fixed. Apple is aware of the issue and working on a fix. In the meantime, I recommend you try to work with XRTouchSpaceInteractor. You can see an example of how this works in the XRI Debug sample scene included in the PolySpatial package samples.

If you are building a VR app, the same constraint applies (you only get the gaze vector on the Began phase of the pinch). However, the data is correct, and can be used to do a raycast. The sample content in the com.unity.xr.visionos package demonstrates how this can be used with XRI as well.

Good luck!

3

any update in 2025? is there any other APIs to use, some inaccurate direction is still ok?

4

No updates. Feel free to request this from Apple via their Feedback Assistant.

5

But gaze works when using the Canvas right? World Canvases do detect where the eye is looking and highlights the correct buttons. What’s the difference here?

6

That doesn’t happen in code that we control; it’s entirely handled through Apple’s software. In RealityKit, we add the HoverEffectComponent to the buttons (along with a CollisionComponent for the collision geometry and the ModelComponent for the visual representation). Our UI shader graph then uses the Hover State node to change the button’s color when hovered (i.e., looked at).

All of this is possible to reproduce from Unity (so that, for example, you can have 3D objects that highlight when gazed at) using:

What is not possible is getting the gaze vector in user code (C#/Swift). That’s simply a limitation that Apple has imposed for all users of their APIs.

7

Hover on world-space uGUI in Metal immersive apps — state in September 2026? →

etup: Unity 6000.0.82f1 · com.unity.xr.visionos 2.4.3, App Mode = Metal Rendering with Compositor Services · XRI 3.3.2 · AR Foundation 6.0.8 · xr.hands 1.7.3 · Input System 1.19.0 · uGUI 2.0.0 · URP 17.0.4 · no PolySpatial. UI is world-space Canvas + Image + TextMeshProUGUI — no MeshRenderers, no colliders.

Still true as far as I know: gaze is never exposed to app code; VisionOSHoverEffect needs a RealityKit entity so it can’t apply in Metal mode; VisionOSSpatialPointerDevice only reports on Began/Moved/Ended, so anything bound to it is pinch-gated.

What changed: visionOS 26 added hover effects to Compositor Services (addTrackingArea + addHoverEffect(.automatic)), same privacy model — which seems to retire the 2024 reason for why Metal apps can’t have hover.

  1. Is there any supported path to pre-pinch hover on world-space uGUI in Metal app mode today? A plain “no” is a useful answer.
  2. Is binding the tracking-area API on the roadmap? And is there a uGUI story, given it needs a per-draw-call index from the fragment shader while uGUI batches?
  3. Concretely: should a head-parented XRRayInteractor with no TrackedPoseDriver drive uGUI hover via XRUIInputModule + TrackedDeviceGraphicRaycaster? Module is active, enableUIInteraction true, but TryGetCurrentUIRaycastResult returns nothing — while from the identical ray a custom BaseRaycaster hits the Button at 1.10 m.
8

Just take the camera rotation, Apple doesn’t allow access to gaze for privacy reason and it’s not going to any time soon.