Is it possible to get hand tracking data from facedown camera?

So we know that Vision Pro has multiple cameras that can track hand data (one example is the facedown camera, which allows you to pinch without raising your arm)

So I’m wondering does XRHand also include hand tracking data from the facedown camera?
From my testing, it looks like XRHand only receive front view camera hand tracking data. If this is true, is it because of some technical restrictions (like maybe even ARKit/RealityKit doesn’t expose the facedown hand tracking data) or is it something that XRHand or PolySpatial can update to support? Or is it just something settings that I didn’t configure properly?

1 Like

On our side (XR Hands) we just report the ARKit hand tracking data from the platform. I imagine all the sensors and camera contribute to hand tracking in some way. As far as I know there isn’t anything special about accessing certain sensors / cameras.

I recommend reaching out to apple or submitting a feedback request for this as it falls more on the platform side.

1 Like

@DanMillerU3D , thanks for the explanation!

One more question if you don’t mind. So Vision OS also reports semantic label with the plane detection, like if a horizontal plane is a floor or a table, etc.

Does Polyspatial or any other Unity package also pass through the semantic label? If not, I’m wondering do you have any suggestion how we can generate similar label on our own? I’m a bit curious if all we know is a plane with width/length/position, how does Vision OS differentiate between a floor and a table?

Yes, you can use either planes (flat surfaces) or a world mesh. Both contain classifications see docs here

If you look at the package samples the mixed reality scene has a small UI that labels each plane classification and alignment. PlaneDataUI.cs

 void OnBoundaryChanged(ARPlaneBoundaryChangedEventArgs eventArgs)
{
       m_ClassificationText.text = m_Plane.classification.ToString();
       m_AlignmentText.text = m_Plane.alignment.ToString();

       transform.position = m_Plane.center;
}
1 Like

This is awesome information, thanks @DanMillerU3D

Guess I need to check the samples/templates more carefully :grin:

1 Like