Handling pointer events on render texture

bdovaz · March 15, 2024, 7:29am

@ericprovencher but can you at least test it? I can attach the project by creating an issue in the issue tracker.

I mean to know if what I pretend is possible somehow, I’m not asking for worldspace support like world space canvas with ugui but support through a render texture which I see is something supported at least without XR… I mean, the 3D UI Toolkit use case with render texture via keyboard and mouse does work since 2021 with @antoine-unity example: https://discussions.unity.com/t/852531/6

Apart from that I have seen this other example (to try something else) but it does not allow me to interact with 2 controllers at the same time:

https://gist.github.com/RoxDevvv/83215ae2fe45c5e7416521fe1697fb03

It is as if the SetScreenToPanelSpaceFunction method is not called when it should and I get in conflict the left and right controls if I use them at the same time.

Thanks.

SeanBannister · March 18, 2024, 6:37am

Consider giving this asset a try, I purchased it but life got it the way and never had a chance to try it out:

bdovaz · March 18, 2024, 7:03am

Thanks, I have already seen that asset but I would like to solve it without a third party paid asset that does more things than I need (curvatures, etc…).

antoine-unity · April 11, 2024, 11:34pm

bdovaz:

@ericprovencher but can you at least test it? I can attach the project by creating an issue in the issue tracker.

I mean to know if what I pretend is possible somehow, I’m not asking for worldspace support like world space canvas with ugui but support through a render texture which I see is something supported at least without XR… I mean, the 3D UI Toolkit use case with render texture via keyboard and mouse does work since 2021 with @antoine-unity example: https://discussions.unity.com/t/852531/6

Apart from that I have seen this other example (to try something else) but it does not allow me to interact with 2 controllers at the same time:

https://gist.github.com/RoxDevvv/83215ae2fe45c5e7416521fe1697fb03

It is as if the SetScreenToPanelSpaceFunction method is not called when it should and I get in conflict the left and right controls if I use them at the same time.

Thanks.

Indeed at the moment the SetScreenToPanelSpaceFunction method is not sufficient to correctly support VR. More logic is needed to correctly interpret input devices on these platforms and send them appropriate UI events.

I imagine it might be possible to make it work by firing events into the UI yourself, but we haven’t tried this ourselves as we are already looking at supporting XR with the actual world space implementation that is progress (which doesn’t require the SetScreenToPanelSpaceFunction override).

bdovaz · April 12, 2024, 5:07am

@antoine-unity but that’s 2 year at least away… I expected some workaround to unlock this use case with 2022 LTS or 2023/6…

“I imagine it might be possible to make it work by firing events into the UI yourself”

Can you explain it more? Thanks.

bdovaz · November 11, 2024, 3:33pm

@antoine-unity it’s been several months since my last post that you didn’t get around to replying to.

I’m really looking forward to removing as much UI as I can from uGUI and redoing it in UIToolkit but if you don’t reply to that post I can’t even try…

csp_xr · November 11, 2024, 4:03pm

Hi there! I am the original poster (I have no longer access to the account which is associated to my university). I initially wrote this for my master’s thesis. Since then things have gone forward, and the thesis is now a (pre-release) extension called XRUI. With the lost account I had totally forgotten about this thread and I never got to thank @antoine-unity for their example which helped me a lot back then! So now I can say thanks in due form

@bdovaz Maybe you can scroll the code to find what you need. It’s been a while since I dived into this, but I think you may in particular be interested by this:

github.com

chwar/XRUI/blob/main/Scripts/XRUIWorldSpaceInteraction.cs

// MIT License
// Copyright (c) 2021 Chris Warin
// Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
// The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
using System;
using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.UIElements;

namespace com.chwar.xrui
{
    /// <summary>
    /// This class interfaces with Unity <see cref="EventHandler"/> and makes XRUI react to world space interactions (e.g., MR/VR pointers). 
    /// </summary>
    public class XRUIWorldSpaceInteraction : MonoBehaviour, IPointerMoveHandler, IPointerUpHandler, IPointerDownHandler,
    ISubmitHandler, ICancelHandler, IMoveHandler, IScrollHandler, ISelectHandler, IDeselectHandler, IDragHandler
    {
        /// <summary>
        /// The <see cref="PanelSettings"/> of the UI that is pointed at.

This file has been truncated. show original

You can also look at the XR Interactions section and acknowledgements section (see gist by katas94) of the README.

With this, you should be able to use UIToolkit with XRI (or any other XR SDK – that being said, I had some weird offset with using the then Oculus SDK with a Quest 2, unsure if that is still the case) and point to your elements with an AR/VR controller. In my projects I managed to use Quest 2 and 3 controllers, and Hololens 2 + hands.

I hope this helps!