Gesture Detection Issues with VisionOS in Unity XR (com.unity.xr.visionos)

Unity Engine
, , , ,
1

I’ve encountered significant issues with gesture detection when using VisionOS in Unity XR. The gesture recognition for the index and middle fingers is reliable, but the pinch gestures involving the ring and little fingers are detected very unreliably. The detection success rate for these gestures is extremely low, around 1-2%, and only in specific hand poses and positions or rotations. The sample project in the com.unity.xr.visionos package also does not detect these gestures reliably.

Steps to Reproduce:

  1. Use the VR Sample (URP) project provided in the com.unity.xr.visionos package.
  2. Observe the hand skeleton and gesture detection in the sample project.
  3. Attempt to perform pinch gestures with the ring and little fingers.

Expected Result: Pinch gestures with the ring and little fingers should be detected as reliably as those with the index and middle fingers.

Actual Result: Pinch gestures with the ring and little fingers are detected very unreliably (1-2% success rate). The hand skeleton in the VisionOS project export does not show these fingers meeting, resulting in failed gesture recognition.

This issue severely limits the usability of gestures involving the ring and little fingers in applications developed with this package. Any assistance or guidance on resolving this issue would be greatly appreciated.

Thank you.

2

Can you confirm what version of XR hands you’re using? There’s a known issue with joint tracking on visionOS that was resolved on version 1.5, which is unfortunately still in pre-release.

Can you give the pre-release version a try if you haven’t already?
https://docs.unity3d.com/Packages/com.unity.xr.hands@1.5/manual/index.html

3

Hi there! Sorry to hear that you’re having trouble.

Could you let us know what specific version of com.unity.xr.visionos you are using as well? We had an issue with this package where we reported joints as not tracked as soon as they were no longer visible to the camera. In the latest versions (1.2.3 and 2.0.0-pre.3) we changed this so that the whole hand always reports as tracked, as long as the wrist is tracked. This should greatly improve gesture recognition. If you aren’t already on the latest version, could you try updating and see if it helps?

Thanks for reaching out, and good luck!