I recently updated XR Interaction Toolkit to the 3.0.3 version where you are describing a new rework of the input architecture managed by Input Readers. Btw, from the documentation I was not able to practically understand what this Input Readers are, and most importantly how are we supposed to use them during development.
In my case, I am working with hand tracking and gestures, and I would like to rebind a classic input action like the “select” one (commonly associated with the pinch hand gesture) in order to be triggered by another gesture, a fist for example. I was searching a way to do this through the Input System in order to let the interactors behave as usual, but I didn’t manage to do this.
Reading the new docs on these Input Readers, I thought it may be my use case, so I would like to have a bit more details about what is the workflow to implement custom ones, and the best practices in using them.
Indeed input readers are perfectly suited to that task.
The gist is that they’re an interface that lets you implement anything you want for the isPerformed book that’s used by interactors to determine if input is active in a given frame.
The input readers are used to implement input for the visionOS sample. I recommend reading some of the code there as an example and reading the [docs](http://Here you go! visionOS Sample | XR Interaction Toolkit | 3.0.4) to understand the intricacies.
In the hand demo scene we also use an input reader to expand the pinch enter and exit thresholds so that grabs are more resilient to occlusion that makes the quest think the pinch ended when it didn’t actually,
Here’s a screenshot from some slides we presented at GDC. Hope they help a bit.
Thanks for the quick response. I was just noticing the implementation of Input Readers into the different XRBaseInteractors in the new XR Rig.
So let me see if I understood correctly. A right way to approach this kind of situation is to write my custom MonoBehaviour implementing IXRInputButtonReader interface, then change the input readers mode of my Interactors to Object Reference and assign that MonoBehaviour as reference. And my custom behaviour should only manage the bool value of ReadIsPerformed property in order to properly work (for example, combining it with the behaviour of your StaticHandGesture component).
Am I missing something?
Also, is there a video about what you showcased at GDC?
What you said sounds good! You’ll also need to ensure you box your updates properly so that the other bools managing that state changes around start and release of the input are properly handled, but you definitely got the gist.
I want to release the videos asap, but it’s on the backlog at the moment on another team to make them ready for public consumption. Hopefully they’ll be live soon.
Hi @ericprovencher ! In these last days I was trying to test my implementation, but since it still doesn’t perform the Select action, I’d like some help understanding if my input reader was setup correctly.
First of all, I simply created my HandGestureDetector component starting from the one available on the package samples (simply more generalized and cleaned up from specific things of the hand gesture sample scene):
public class HandGestureDetector : MonoBehaviour
{
[SerializeField]
[Tooltip("The hand tracking events component to subscribe to receive updated joint data to be used for gesture detection.")]
XRHandTrackingEvents _handTrackingEvents;
[SerializeField]
[Tooltip("The hand shape or pose that must be detected for the gesture to be performed.")]
ScriptableObject _handShapeOrPose;
[SerializeField]
[Tooltip("The minimum amount of time the hand must be held in the required shape and orientation for the gesture to be performed.")]
float _minimumHoldTime = 0.2f;
[SerializeField]
[Tooltip("The interval at which the gesture detection is performed.")]
float _gestureDetectionInterval = 0.1f;
[Tooltip("The event fired when the gesture is performed.")]
public UnityEvent gesturePerformed;
[Tooltip("The event fired when the gesture is ended.")]
public UnityEvent gestureEnded;
XRHandShape _handShape;
XRHandPose _handPose;
bool _wasDetected;
bool _performedTriggered;
float _timeOfLastConditionCheck;
float _holdStartTime;
public Handedness Handedness => _handTrackingEvents.handedness;
private void OnEnable()
{
_handTrackingEvents.jointsUpdated.AddListener(OnJointsUpdated);
_handShape = _handShapeOrPose as XRHandShape;
_handPose = _handShapeOrPose as XRHandPose;
}
private void OnDisable()
{
_handTrackingEvents.jointsUpdated.RemoveListener(OnJointsUpdated);
}
private void OnJointsUpdated(XRHandJointsUpdatedEventArgs eventArgs)
{
if (!isActiveAndEnabled || Time.timeSinceLevelLoad < _timeOfLastConditionCheck + _gestureDetectionInterval)
return;
var detected =
_handTrackingEvents.handIsTracked &&
_handShape != null && _handShape.CheckConditions(eventArgs) ||
_handPose != null && _handPose.CheckConditions(eventArgs);
if (!_wasDetected && detected)
{
_holdStartTime = Time.timeSinceLevelLoad;
}
else if (_wasDetected && !detected)
{
_performedTriggered = false;
gestureEnded?.Invoke();
}
_wasDetected = detected;
if(!_performedTriggered && detected)
{
var holdTimer = Time.timeSinceLevelLoad - _holdStartTime;
if(holdTimer > _minimumHoldTime)
{
gesturePerformed?.Invoke();
_performedTriggered = true;
}
}
_timeOfLastConditionCheck = Time.timeSinceLevelLoad;
}
}
Then, I wrote my own HandGestureInputReader component implementing the IXRInputButtonReader interface. This is where I managed all the bools required. I tested them and they seem to work as intended. Here’s the code:
public class HandGestureInputReader : MonoBehaviour, IXRInputButtonReader
{
private class BoolReference
{
public bool Value { get; set; }
public BoolReference(bool reference)
{
Value = reference;
}
}
[SerializeField] private HandGestureDetector _gestureDetector;
bool _currentlyPerformed, _frameCheck;
BoolReference _wasDetectedThisFrame, _wasReleasedThisFrame;
private void OnEnable()
{
_wasDetectedThisFrame = new BoolReference(false);
_wasReleasedThisFrame = new BoolReference(false);
_gestureDetector.gesturePerformed.AddListener(OnGesturePerformed);
_gestureDetector.gestureEnded.AddListener(OnGestureEnded);
}
private void OnDisable()
{
_gestureDetector.gesturePerformed.RemoveListener(OnGesturePerformed);
_gestureDetector.gestureEnded.RemoveListener(OnGestureEnded);
}
private void OnGesturePerformed()
{
_currentlyPerformed = true;
}
private void OnGestureEnded()
{
_currentlyPerformed = false;
}
private void Update()
{
if (_currentlyPerformed && !_frameCheck)
StartCoroutine(TrueForOneFrame(_wasDetectedThisFrame));
else if (!_currentlyPerformed && _frameCheck)
StartCoroutine(TrueForOneFrame(_wasReleasedThisFrame));
_frameCheck = _currentlyPerformed;
}
private IEnumerator TrueForOneFrame(BoolReference boolRef)
{
boolRef.Value = true;
yield return new WaitForEndOfFrame();
boolRef.Value = false;
}
// Interface Methods
public bool ReadIsPerformed()
{
return _currentlyPerformed;
}
public float ReadValue()
{
return _currentlyPerformed ? 1 : 0;
}
public bool ReadWasCompletedThisFrame()
{
return _wasReleasedThisFrame.Value;
}
public bool ReadWasPerformedThisFrame()
{
return _wasDetectedThisFrame.Value;
}
public bool TryReadValue(out float value)
{
value = _currentlyPerformed ? 1 : 0;
return _currentlyPerformed;
}
}
Finally, my last component is a helper tool called XRInteractorInputOverride, which simply substitute the Select or the Activate action Input Reader on the specified XR Interactor:
As I mentioned, the logic behind the HandGestureInputReader seems to work as intended (bools are true when they have to), so can you help me understand why my Near-Far Interactor still does not trigger Select action when the gesture is detected?
First, have you tried just assigning the input reader in the editor? You’re meant to set it “object” and then slide the reference in.
If it’s still failing, the reason it’s probably failing is because you’re updating the gesture value whenever the hand joint events update and then updating the other books with coroutine on potentially different frames. All 3 bools should be updated in the same frame ideally.
Here’s what I recommend. Have a queue for state changes and only process one per frame in your input reader. Just listen for is performed.
In an update loop in that class, pop the queue and then do this
update performed ( bool newIsPerformed)
m_WasPerformedThisFrame = !
Thanks for the help! Seems like the coroutine approach was the problem. Thanks to your hint I was able to simplify the logic and the version below seems to work:
Thank you! Just an advice, I watched it, but I think it doesn’t cover a proper example on how to actually deal with Input Readers and how to manage this new architecture for our own purpose. With the clarification you gave me I was able to understand, but maybe some kind of content more dedicated to Input Readers themselves (more “practical” than the documentation) may be useful to the community!
Thank you for your extremely quick response. I read the documentation and what HunterProduction did but I didn’t fully understand yet. My main goal is to create new input action based on gestures created with XR Hand, how exactly can I achieve that with this new input reader? thank you for the clarification.
thank you for the clarification.