Hands Gestures recognition

Unity Engine
, ,
1

Hi, I’m working on Hands Inputs for the Vision. I did check the hands package and found out that there are 6 different hands gestures, but looks like it’s just predefined Poses of each joints. I was wondering do you have plans to have gesture recognition code in the future, and if yes maybe you have rough estimation for that. I did find DetectPinch function. So it would be nice to have “DetectFist” etc. functions as well.

2

Yes a gesture detection system is being worked on by the larger XR team. It uses finger Curl, Extension, Flex, Pinch, Splay values combined with hand orientation. It will be available in a future hands package.

I don’t have an estimation of when that work will land.

3

Thanks, for the response, do you have information about what will be in the next update of the Vision OS beta?

4

XR Hands new gesture recognition works fine!

5

Hi Dan, do we have access to higher level gesture recognition packages right now? Like detect fist, detect palm flex etc?
If that’s not supported right now, any third party plug-ins we can use to achieve this?

6

Hi Antonio do you have any tutorial I can find to follow on hand gesture recognition?

7

Hey there! Sorry for the late notice on this one, but we ended up shipping a fix for this issue in the 1.2.3 version of com.unity.xr.visionos. Essentially, as we all came to realize in this and other threads, visionOS reports is_tracked = false for joints that are not visible, but for which there still exists up-to-date pose data. We now always report every joint as being tracked as far as XR Hands is concerned, as long as the “hand anchor” is tracked, meaning at least the wrist is visible.

This should make gesture detection and general interactions more reliable as the transforms for finger joints will still exist and move naturally even as they are occluded. There is also a new VisionOSHandExtensions API to query the is_tracked state in case you want to know whether the joint is occluded. The updated sample uses this to swap in a semi-transparent prefab for occluded joints to show a visible difference between the two.

Thanks again for reporting this issue, and please let us know if you were able to update to the latest version of the packages and try it out.

1 Like