Hello.
I want to implement a rotation operation using scripting to get the input with both hands that is displayed by pressing the option key on the simulator.
I think I can do this if I can get the position of each of the two hands.
Thanks in advance.
Hey there! You can do this using SpatialPointerDevice. If you take a look at the InputDataVisualization scene in the PolySpatial Samples, it shows how to get input from both hands at the same time.
Hope this helps!
I see, we can use two input values, Touch[0] and Touch[1]!
Thanks for the advice!
var touches = Touch.activeTouches;
if (touches.Count >= 1)
{
var zeroPos = EnhancedSpatialPointerSupport.GetPointerState(touches[0]).devicePosition;
}
if(touches.Count >= 2)
{
var onePos = EnhancedSpatialPointerSupport.GetPointerState(touches[1]).devicePosition;
}
Based on your advice, I was able to get the positions of both hands as described above.
It seems that the coordinate system of zeroPos and onePos are different, is there a way to handle them in the same coordinate system?
Or is there a wrong way to get them?
Do you mean they are different from each other, or from the expected coordinates you get out of interactionPosition?
Are you testing in the simiulator or on a device? In the simulator, the positions can end up looking kind of weird because they are placed near the camera, and end up translating quite a bit as you move the mouse around. Unfortunately this is just a quirk of the visionOS simulator which we can’t do anything about on the Unity side. If you test on device, the device position should show up at the location of your pinched fingers.
If you are having trouble testing your app’s interactions in the visionOS simulator, I encourage you to share your feedback with Apple (using the Feedback Assistant on macOS or Help > Send Simulator Feedback in the simulator app).
I am running this on the visionOS simulator.
When the white circle displayed at the mouse position overlaps the other white circle displayed when the option key is pressed, I thought the devicePosition of the two circles would have the same value, but in fact they have different values.
The interactionPosition also had a different value.
Is this a problem unique to the visionOS simulator, or do the two values match in the actual device?
Each of the touches represents a different hand, so I would not expect them to have the same value.
For example, if you first pinch with your left hand and then pinch with your right hand, touches[0] will represent your left hand and touches[1] will represent your right hand. The interactionPosition for touches[0] will start out at where your gaze vector intersects an object, in other words where you were looking. As you move your left hand, the interactionPosition for touches[0] will move around based on this hand motion. Bear in mind that the platform will apply some “acceleration” to this movement depending on how close or far away the object was when you started the interaction.
The devicePosition of touches[0] will represent the location of your pinched fingers on your left hand, and will follow that motion directly one-to-one. If you move your hand 10cm, the devicePosition should move 10cm.
touches[1] will contain data representing your right hand. Thus, I would expect that maybe the interactionPosition value for touches[1] is close to the same value as interactionPosition for touches[0], but the devicePosition for touches[0] is probably further away from devicePosition for touches[1] because your hands wouldn’t have been in the same location.
Again, all of this can look a little funny in the simulator, because it is approximating all of this with a mouse, and doesn’t track your hands like the device will. Do you have access to a device? This might make more sense if you can test it out on physical hardware.
I see, so the white circle on the visionOS simulator represents the position of the eye, not the position of the hand.
Unfortunately, the device will not be available until next March or later, so I will try again when it is available.
My question is, are gestures like zooming and rotating at 12:42 in the video on the page below supported by PolySpatial?
Kind of 
As far as I can tell, the white circles on the simulator window are a holdover from the iOS simulator, where they represent the location on screen where the simulated users’ fingers would be pressing. I think on the visionOS simulator, they represent the origin point for the simulated gaze ray, in other words, they represent where you would have clicked to start one of the two interactions. However, I haven’t found that theory to be particularly consistent. My advice would be to ignore the circles themselves and pay attention to where the interactionPosition ends up in 3D space.
Yes. Two-handed interactions like pinch-zoom are supported, although you will need to implement your own logic for how the two individual device positions result in a 3D transformation. This is something we will try to incorporate in future visionOS samples, but you may find some helpful examples in the XR Interaction Toolkit or the MR/VR templates, which currently support Meta Quest and other XR platforms with hand tracking.
As far as I can tell, the white circles on the simulator window are a holdover from the iOS simulator, where they represent the location on screen where the simulated users’ fingers would be pressing.
I thought it was a feature of visionOS!
I will try not to worry too much about this white circle.
This is something we will try to incorporate in future visionOS samples, but you may find some helpful examples in the XR Interaction Toolkit or the MR/VR templates, which currently support Meta Quest and other XR platforms with hand tracking.
I will refer to those samples.
Thank you for answering my many questions!
Hi, @mnakagami @mtschoen, how to achieve hand gesture scaling and rotation in videos based on template 1.1.6 samples?
I feel that the status of Touch [0] and Touch [1] is not determined every frame.
Here is my key code, mainly focusing on hand scaling. Can you give me some suggestions?
Thank you very much!
if (activeTouches.Count == 1 && !isPinching)
{
var primaryTouch0Data = EnhancedSpatialPointerSupport.GetPointerState(activeTouches[0]);
var touch0Phase = activeTouches[0].phase;
if (touch0Phase == TouchPhase.Began && primaryTouch0Data.Kind == SpatialPointerKind.IndirectPinch || primaryTouch0Data.Kind == SpatialPointerKind.DirectPinch)
{
if (primaryTouch0Data.targetObject != null)
{
if (primaryTouch0Data.targetObject.TryGetComponent(out FbxObjectBehaviour boundedObject))
{
Debug.Log("primaryTouchData.targetObject 1-0 = " + primaryTouch0Data.targetObject);
m_SelectedObject0 = boundedObject;
m_SelectedObject0.Select(true);
if (m_AudioSource != null && !m_ActiveDirectPinch)
{
m_AudioSource.volume = k_StartAudioVolume;
m_AudioSource.Play();
m_RealTimeAtPinch = Time.realtimeSinceStartup;
m_PlayedEndAudio = false;
}
m_ActiveDirectPinch = primaryTouch0Data.Kind == SpatialPointerKind.DirectPinch;
initial0Pos = primaryTouch0Data.interactionPosition;
}
}
}
if (touch0Phase == TouchPhase.Moved)
{
if (m_SelectedObject0 != null)
{
m_SelectedObject0.MoveDirectly(primaryTouch0Data);
initial0Pos = primaryTouch0Data.interactionPosition;
}
}
if (touch0Phase == TouchPhase.Ended || touch0Phase == TouchPhase.Canceled)
{
if (m_SelectedObject0 != null)
{
m_SelectedObject0.Select(false);
m_SelectedObject0 = null;
m_ActiveDirectPinch = false;
m_PlayedEndAudio = true;
}
}
}
else if (activeTouches.Count == 2)
{
var primaryTouch0Data = EnhancedSpatialPointerSupport.GetPointerState(activeTouches[0]);
var touch0Phase = activeTouches[0].phase;
var primaryTouch1Data = EnhancedSpatialPointerSupport.GetPointerState(activeTouches[1]);
var touch1Phase = activeTouches[1].phase;
if (touch0Phase == TouchPhase.Began && primaryTouch0Data.Kind == SpatialPointerKind.IndirectPinch || primaryTouch0Data.Kind == SpatialPointerKind.DirectPinch)
{
Debug.Log("primaryTouchData.targetObject 2-0 = " + primaryTouch0Data.targetObject);
initial0Pos = primaryTouch0Data.interactionPosition;
}
if (touch1Phase == TouchPhase.Began && primaryTouch1Data.Kind == SpatialPointerKind.IndirectPinch || primaryTouch1Data.Kind == SpatialPointerKind.DirectPinch)
{
Debug.Log("primaryTouchData.targetObject 2-1 = " + primaryTouch1Data.targetObject);
initial1Pos = primaryTouch1Data.interactionPosition;
}
if (!isPinching)
{
// Start zooming when the second finger touches the screen
isPinching = true;
initialDistance = Vector3.Distance(initial0Pos, initial1Pos);
if (m_SelectedObject0 != null)
{
initialScale = m_SelectedObject0.RootTransform.localScale;
}
}
if (touch0Phase == TouchPhase.Moved || touch1Phase == TouchPhase.Moved)
{
if (m_SelectedObject0 != null)
{
Vector3 cur0Pos = primaryTouch0Data.interactionPosition;
Vector3 cur1Pos = primaryTouch1Data.interactionPosition;
// Calculate new distance and scaling ratio
float currentDistance = Vector3.Distance(cur0Pos, cur1Pos);
float scaleFactor = currentDistance / initialDistance;
if (isPinching)
{
// Apply Zoom
m_SelectedObject0.RootTransform.localScale = initialScale * scaleFactor;
}
}
}
if (touch0Phase == TouchPhase.Ended || touch0Phase == TouchPhase.Canceled)
{
if (m_SelectedObject0 != null)
{
m_SelectedObject0.Select(false);
m_SelectedObject0 = null;
m_ActiveDirectPinch = false;
m_PlayedEndAudio = true;
}
}
}
else if (activeTouches.Count < 2)
{
// Stop zooming when the number of touches is less than two
isPinching = false;
}
It may be better to use inputDevicePosition instead of interactionPosition, as the actual hand position can be obtained.
I also noticed that when I pinched both hands together, the TouchPhase of both hands was not being detected correctly, so I removed as much code as possible that relied on TouchPhase and focused on the information held by the SpatialPointerState.
It’s much more practical to utilize whether each hand is holding a valid target after pinching, and if so, what the tag is, what the SpatialPointerKind state is, etc.
else if (activeTouches.Count == 2)
{
SpatialPointerState firstTouchData = EnhancedSpatialPointerSupport.GetPointerState(activeTouches[0]);
TouchPhase touchPhase = activeTouches[0].phase;
SpatialPointerState secondTouchData = EnhancedSpatialPointerSupport.GetPointerState(activeTouches[1]);
TouchPhase secondTouchPhase = activeTouches[1].phase;
if (m_unboundFirst == null && m_unboundSecond == null &&
firstTouchData.targetObject != null && firstTouchData.Kind == SpatialPointerKind.IndirectPinch)
{
if (firstTouchData.targetObject.TryGetComponent(out UnboundedObjectBehavior unboundFirst))
{
m_unboundFirst = unboundFirst;
HandleSingleHandDoubleTap(touchPhase, firstTouchData);
}
}
if (m_unboundSecond == null && secondTouchData.targetObject != null && secondTouchData.Kind == SpatialPointerKind.IndirectPinch)
{
if (secondTouchData.targetObject.TryGetComponent(out UnboundedObjectBehavior unboundSecond))
{
m_unboundSecond = unboundSecond;
if (m_unboundFirst == null)
{
HandleSingleHandDoubleTap(touchPhase, firstTouchData);
}
else
{
m_PrePinchTime = 0f;
}
}
}
if (m_unboundFirst != null && m_unboundSecond != null && touchPhase == TouchPhase.Moved && secondTouchPhase == TouchPhase.Moved)
{
if (m_unboundFirst.tag == m_unboundSecond.tag && firstTouchData.Kind == SpatialPointerKind.IndirectPinch &&
secondTouchData.Kind == SpatialPointerKind.IndirectPinch)
{
Vector3 firstTouchInputPos = firstTouchData.inputDevicePosition;
Vector3 secondTouchInputPos = secondTouchData.inputDevicePosition;
if (m_FirstPrePoint != Vector3.zero)
{
//Working with zoom or rotation.
}
m_FirstPrePoint = firstTouchInputPos;
m_SecondPrePoint = secondTouchInputPos;
}
m_PrePinchTime = 0f;
}
if (touchPhase == TouchPhase.Ended || touchPhase == TouchPhase.Canceled ||
secondTouchPhase == TouchPhase.Ended || secondTouchPhase == TouchPhase.Canceled)
{
m_unboundFirst = null;
m_unboundSecond = null;
Debug.Log($"{Time.time}] End!!!");
}
}