XRHands mesh issues

visionOS Beta Virtual Reality (Fully Immersive) Apps
1

Hi, I’m working on a fully immersive app using the built-in render pipeline. I’ve recently brought the XRHands package’s sample into my test project so I can visualize the hand data. When I run on device, the hand meshes appear crinkled and bent in a very strange way. Is this expected to work, or does the device have joints formatted for a different rig? At this time, is the best way to look at the data to simply visualize the joint positions yourself? They also draw weirdly behind the passthrough hands visualization. Is there a way to draw on top of the passthrough hands? Perhaps by mucking with the render queue values in the shader? Thanks!

Mike

thanks,

Mike

2

I’ll also note that the built in joints visualization (checking “DebugDrawJoints” on the HandVisualizer script does not produce anything visible, even with the passthrough hands turned off… Anyone get this to work?

3

My understanding is that this is correct. The rotation space for the hand joints is different from what the XR Hand package normally returns and I had to update my hand tracking code to support it. @DanMillerU3D briefly discusses this in his Unity visionOS Beta Program video starting about 5:42. This might be fixed up in the future as mentioned by Dan Miller in this thread: XRHands Package - AVP ARKit Hand Tracking Data Fixup - #3 by OwlchemyDawson

That seems like the best for me and is what I do in my hand tracking system.

At a recent Apple Developer Lab we asked the same question about the hands and were told by an Apple engineer that it is not possible to render over top of the upper limbs due to the way the compositor works – composites passthrough image followed by the application’s image, and finally the upper limb cutout (all at the compositor level outside of Unity purview). However, we also learned that you can turn off the default functionality for showing the upper limbs by tweaking the UnityMain.swift in the generated Xcode project. I made a post here with the details and requested a setting in the Unity Editor to control this.

2 Likes
4

Thanks (as always) for the through details, this is saving me a lot of time!

So that chart that Dan Miller shows in the video you linked has the mapping of the actual joints to the corresponding int in Unity’s XRHandJointID enum? Here’s the chart to in case it helps anyone else:

arkit_to_xrjointid
arkit_to_xrjointid1899×951 140 KB

Bummer about not being able to layer visuals on top of the passthrough hands…

1 Like
5

@puddle_mike did you get an answer to this mapping of joints?

6

That diagram I posted didn’t turn out to be fully correct from what I’ve observed. Through some debug visualization, I’ve come to the following mapping between our internal enum for each joint label and the values we get from XR Hands:

XRHandJointID _ToXRHandJoint(SNHandJoint j)
   {
      //this mapping is essentially from visual debugging
      switch (j)
      {
         case SNHandJoint.Wrist: return (XRHandJointID)1;
         case SNHandJoint.Thumb0: return XRHandJointID.EndMarker; //NOT SUPPORTED
         case SNHandJoint.Thumb1: return (XRHandJointID)3;
         case SNHandJoint.Thumb2: return (XRHandJointID)4;
         case SNHandJoint.Thumb3: return (XRHandJointID)5;
         case SNHandJoint.ThumbTip: return (XRHandJointID)6;
         case SNHandJoint.Index1: return (XRHandJointID)7;
         case SNHandJoint.Index2: return (XRHandJointID)8;
         case SNHandJoint.Index3: return (XRHandJointID)9;
         case SNHandJoint.IndexIntermediateTip: return (XRHandJointID)10;
         case SNHandJoint.IndexTip: return (XRHandJointID)11;
         case SNHandJoint.Middle1: return (XRHandJointID)12;
         case SNHandJoint.Middle2: return (XRHandJointID)13;
         case SNHandJoint.Middle3: return (XRHandJointID)14;
         case SNHandJoint.MiddleIntermediateTip: return (XRHandJointID)15;
         case SNHandJoint.MiddleTip: return (XRHandJointID)16;
         case SNHandJoint.Ring1: return (XRHandJointID)17;
         case SNHandJoint.Ring2: return (XRHandJointID)18;
         case SNHandJoint.Ring3: return (XRHandJointID)19;
         case SNHandJoint.RingIntermediateTip: return (XRHandJointID)20;
         case SNHandJoint.RingTip: return (XRHandJointID)21;
         case SNHandJoint.Pinky0: return XRHandJointID.EndMarker; //NOT SUPPORTED
         case SNHandJoint.Pinky1: return (XRHandJointID)22;
         case SNHandJoint.Pinky2: return (XRHandJointID)23;
         case SNHandJoint.Pinky3: return (XRHandJointID)24;
         case SNHandJoint.PinkyIntermediateTip: return (XRHandJointID)25;
         case SNHandJoint.PinkyTip: return (XRHandJointID)26;
         case SNHandJoint.ForearmWrist: return (XRHandJointID)1; //i think this is redundant with Wrist
         case SNHandJoint.ForearmArm: return XRHandJointID.EndMarker; //ran outta numbers?!
         default: return XRHandJointID.EndMarker;
      }
1 Like
7

The mapping seems to be fixed to conform with the standard (based on just a couple of joints I tested) in XR.VisionOS v 0.5.0.
Basically I used your example puddle_mike and switched the mapping back to the commented out one. Thank you!

8

Oh great, thanks for the heads up!