Interact with UI button through Render Texture

Unity Engine
1

I’m looking for a solution on how to interact with UI buttons and scrolls through a Render Texture.

I have found this example, and it works great for objects like cubes and spheres, but it doesn’t work with UI.

Hopefully someone knows how to implement this with a UI canvas.

2 Likes
2

I know it is a super long shot but… did you ever figure this out? I have a setup that because of certain pixel processing requires a stacked camera be sent through a render texture before final output. I NEED the UI to exist behind the render texture.

Basically I have a UICAM which renders the UI, it is in the stack with the Base “GameCam”. GameCam renders to a render texture and then the RenderCam is an orthographic camera pointing to a plane with the render texture applied to it.

I KNOW its a scuffed setup but I can’t find another way to both force low resolution pixel perfection on a UI, while ALSO preventing that UI from receiving post processing effects.

3 
....
private bool PointerInImage()
    {
        if (targetGraphic == null)
        {
            return false;
        }

        //get mouse pos
        Vector3 mv = Camera.main.ScreenToWorldPoint(MouseScreenPosition);
        //get world rect
        Rect worldRect = GetWorldRect(targetGraphic.rectTransform);

        //if mouse is in rect, find relative position in rect and compare to targetTexture
        if (worldRect.Contains(mv))
        {
            Texture2D tex = targetGraphic.mainTexture as Texture2D;
            float x = FindRelativePercentage(mv.x, worldRect.xMin, worldRect.xMax);
            float y = FindRelativePercentage(mv.y, worldRect.yMin, worldRect.yMax);

            //Multiply by texture dimensions to get position relative to texture
            x *= tex.width;
            y *= tex.height;

            //check if relative pos falls on non-transparent texture coord
            Color c = tex.GetPixel((int)x, (int)y);
            if (c.a > 0)
            {
                //If so then our pointer is over the targetGraphic
                return true;
            }
        }
        return false;
    }
   
    //Returns a value between 0 and 1 which is the percentage x is between a and b
    private float FindRelativePercentage(float x, float a, float b)
    {
        return (x - a) / (b - a);
    }

    //Returns a Rect relative to world coordinates. Very useful!
    private Rect GetWorldRect(RectTransform rectTransform)
    {
        Vector3[] corners = new Vector3[4];
        rectTransform.GetWorldCorners(corners);
        // Get the bottom left corner.
        Vector3 position = corners[0];

        Vector2 size = new Vector2(
            rectTransform.lossyScale.x * rectTransform.rect.size.x,
            rectTransform.lossyScale.y * rectTransform.rect.size.y);

        return new Rect(position, size);
    }

This function will tell you if the mouse is currently over a targetGraphic, which can be a texture2D, or texture from a button (in which case you’ll need to cast it to tex2D) also make sure to enable read-write of the texture in texture import settings. I’m sure there’s a way to get around that but that’s for you to figure out. If you extend the button, you can call the EventSystem methods OnPointerDown, Up, Enter, Exit. You can also use EventSystem.current.IsPointerOverGameObject() if you want the button to behave like a unity button raycasting-wise.Though it’s a let response, hopefully, it can help anyone in the future!

4

For anyone seeing this:
This code won’t really work in 3D, as it’s comparing a point in 3D with a rect in 2D. For this to work in 3D, it would need to intersect a ray in 3D with a rect in 3D.

I’m currently working on an alteration that does work in 3D, but I may end up doing the smart thing instead and just passing my raycasts from one camera to the next. There’s a lot of support online for that sort of thing, because the same techniques can be used for in-game objects like phones and TVs as the post-processing techniques that Shinobi1507 and I are using.

(Currently I’m using a rather stupid setup where I change the render target of the camera twice each frame, so I can use the same camera to render to a texture and to a screen. It was quick to setup, but it’s caused a lot of problems.)

5

I’d recommend this solution for anyone dealing with this issue. It’s very grab-and-go, and much more versatile than manually checking for collisions yourself.