Hi, I’m working on a fully immersive app using the built-in render pipeline. I’ve recently brought the XRHands package’s sample into my test project so I can visualize the hand data. When I run on device, the hand meshes appear crinkled and bent in a very strange way. Is this expected to work, or does the device have joints formatted for a different rig? At this time, is the best way to look at the data to simply visualize the joint positions yourself? They also draw weirdly behind the passthrough hands visualization. Is there a way to draw on top of the passthrough hands? Perhaps by mucking with the render queue values in the shader? Thanks!
I’ll also note that the built in joints visualization (checking “DebugDrawJoints” on the HandVisualizer script does not produce anything visible, even with the passthrough hands turned off… Anyone get this to work?
That seems like the best for me and is what I do in my hand tracking system.
At a recent Apple Developer Lab we asked the same question about the hands and were told by an Apple engineer that it is not possible to render over top of the upper limbs due to the way the compositor works – composites passthrough image followed by the application’s image, and finally the upper limb cutout (all at the compositor level outside of Unity purview). However, we also learned that you can turn off the default functionality for showing the upper limbs by tweaking the UnityMain.swift in the generated Xcode project. I made a post here with the details and requested a setting in the Unity Editor to control this.
Thanks (as always) for the through details, this is saving me a lot of time!
So that chart that Dan Miller shows in the video you linked has the mapping of the actual joints to the corresponding int in Unity’s XRHandJointID enum? Here’s the chart to in case it helps anyone else:
That diagram I posted didn’t turn out to be fully correct from what I’ve observed. Through some debug visualization, I’ve come to the following mapping between our internal enum for each joint label and the values we get from XR Hands:
XRHandJointID _ToXRHandJoint(SNHandJoint j)
//this mapping is essentially from visual debugging
case SNHandJoint.Wrist: return (XRHandJointID)1;
case SNHandJoint.Thumb0: return XRHandJointID.EndMarker; //NOT SUPPORTED
case SNHandJoint.Thumb1: return (XRHandJointID)3;
case SNHandJoint.Thumb2: return (XRHandJointID)4;
case SNHandJoint.Thumb3: return (XRHandJointID)5;
case SNHandJoint.ThumbTip: return (XRHandJointID)6;
case SNHandJoint.Index1: return (XRHandJointID)7;
case SNHandJoint.Index2: return (XRHandJointID)8;
case SNHandJoint.Index3: return (XRHandJointID)9;
case SNHandJoint.IndexIntermediateTip: return (XRHandJointID)10;
case SNHandJoint.IndexTip: return (XRHandJointID)11;
case SNHandJoint.Middle1: return (XRHandJointID)12;
case SNHandJoint.Middle2: return (XRHandJointID)13;
case SNHandJoint.Middle3: return (XRHandJointID)14;
case SNHandJoint.MiddleIntermediateTip: return (XRHandJointID)15;
case SNHandJoint.MiddleTip: return (XRHandJointID)16;
case SNHandJoint.Ring1: return (XRHandJointID)17;
case SNHandJoint.Ring2: return (XRHandJointID)18;
case SNHandJoint.Ring3: return (XRHandJointID)19;
case SNHandJoint.RingIntermediateTip: return (XRHandJointID)20;
case SNHandJoint.RingTip: return (XRHandJointID)21;
case SNHandJoint.Pinky0: return XRHandJointID.EndMarker; //NOT SUPPORTED
case SNHandJoint.Pinky1: return (XRHandJointID)22;
case SNHandJoint.Pinky2: return (XRHandJointID)23;
case SNHandJoint.Pinky3: return (XRHandJointID)24;
case SNHandJoint.PinkyIntermediateTip: return (XRHandJointID)25;
case SNHandJoint.PinkyTip: return (XRHandJointID)26;
case SNHandJoint.ForearmWrist: return (XRHandJointID)1; //i think this is redundant with Wrist
case SNHandJoint.ForearmArm: return XRHandJointID.EndMarker; //ran outta numbers?!
default: return XRHandJointID.EndMarker;
The mapping seems to be fixed to conform with the standard (based on just a couple of joints I tested) in XR.VisionOS v 0.5.0.
Basically I used your example puddle_mike and switched the mapping back to the commented out one. Thank you!