In our fully immersive app (i.e not using Polyspatial), we need to know when a pinch is being “held” with either hand. We implement this by looking at the “phase” of each VisionOSSpatialPointerState input (seeing if its “Begin” or “Moved”) and assigning a hand by comparing skeletal tracking data to its “inputDevicePosition” property.
This has been working pretty well for a while, but occasionally we get “stuck” thinking a hand is pinching once the pinch has been released. So far as I can tell, if we keep polling the VisionOSSpatialPointerState, it reports a “Begin” or “Moved” phase even tho the fingers have long since separated. The pattern I see the next time the fingers pinch is a phase of “None” followed by “Moved” about a frame later, and receiving “Ended” as expected when the fingers separate.
We can sorta work around this by looking at the skeletal data and “unsticking” our state when the fingers separate enough, but would prefer not to put in a hack like that. Is the behavior I described above something anyone else has seen? Am I just doing this wrong?! Is it a bug on the Unity side of things? @mtschoen any idea?
In the repro project, a cube will turn green when we think the left hand is pinching, and another when we think the right hand is pinching. You can find the pinch detection approach I described in the SNHandTracking_XRHands._RefreshPinchDetector() function.
Thanks for the repro case! Have you tried listening for the events in actionPerformed? This user suggested an event-based approach, which may be the best way to work around this issue on your end at the moment.
The fundamental issue here is that the OS sends events to us regardless of when they occur within a given Unity frame. So it’s possible to get multiple events on the same frame (for example, move, ended, began) or go a few frames without any events coming through. It turns out that when you poll for input state when events come through in this way, you get some weird behavior. It may be easier for you to implement actionPerformed and queue events in your own script so that you can play them back in the right order and “catch up” on any events you would otherwise miss polling the state in Update.
We’re working on some samples for this, and I’m trying to see if there’s a better way to integrate this input with the input system so that polling is more reliable.
Let me know if you have any questions about this approach, or if you’re still running into trouble. Good luck!
Thanks for the tips, I’ll try the event based approach and let you know how it goes. It is curious tho, since if there are events I’m “skipping” over, you’d think it would eventually settle into a phase of “None” (which I detect as “not pinching”, i.e I don’t rely on seeing the “Ended” phase…). I’ll report back soon tho!
@mtschoen - I updated to an event based approach, but can still get the pinch state stuck on. With enough quick pinches in my test app, eventually it gets left with a phase of Begin or Moved… I can share the updated event-based code, but seems to behave pretty much the same
At the top of SNHandTracking_XRHands.cs, there is a variable you can set to use the polling approach or the event based one: see “USE_EVENT_BASED_PINCH_DETECTOR”
Hey @puddle_mike. I observed this issue as well. My solution was to add a timeout where I’d end select interactions if no new event comes in for a timeout period, while my internal select tracking is set to true.
It’s not an ideal fix, but I suspect there’s something weird going on with input and Volumes in Apple’s OS, and I don’t think Unity’s stack is doing anything wrong. I haven’t observed this exact issue when using unbounded mode.
Thanks for the suggestion @ericprovencher! Unfortunately our game has a mechanic where you can hold the pinch as long as you want, so a timeout won’t work for us. At the moment, my workaround is to look at the thumb and index finger tips via skeletal tracking and release the pinch if they separate enough for a few frames. I don’t love the solution since I’m certain to not have the threshold tuned right for all users (and if I get too aggressive, it risks false positives).
Does anyone know what native API’s are used under the hood? If we suspect it’s an issue on Apple’s side, if I’m able to create a native repro, I’ll likely get more traction with them. Not sure my swift skills are up to snuff, but I’ll give it a try
Hey Mike ;-> We’re having the same issue with secondary pinch holds in FLY and using your workaround, just wondering if you ever found a real solution. The problem also occurs when the primary pointer unpinches while the secondary pointer is still pinched. That it’s every time with the first unpinch and not afterward suggests broken logic in the AVP input system not an event queue issue.
Hey! Unfortunately, we haven’t found a real solution yet. We actually disabled the workaround temporarily as well, because we were seeing some occasional false positives. We probably see the pinch stick 1 or 2 times in a 5-10min play session (tho worse on occasion). You end up having to pinch again for it to “release”…
We are going to get back to this issue soon. The best path to get help directly from Apple would be a standalone repro using just native APIs, so this is still on our TODO list…
You mentioned “secondary pinch holds”, and I’ll note that our game is controllable with one hand, so we don’t run into a lot of cases involving both hands. We did do some limited testing with overlapping pinches on both hands, and didn’t see any additional issues yet…
Have you tried on VisionOS2? We haven’t been brave enough to update yet, but should be able to check it out soon to see if there are any improvements.
