Raycasting and Render Texture

Hello everyone.

I have a static camera looking at a “TV screen”.

The “TV screen” is a model and a render texture showing a second camera.

An object is moving in front of the second camera (think surveillance camera)

I want to be able to click the object trough the render texture.

Any ideas of how it can be done?

Thanks!

First perform a raycast, that if it hits the plane/mesh of your TV screens render texture, get the texture coordinate of the hit. Use this texture coordinate to create a ray from the second camera using ViewportPointToRay. Feed the texture coordinate from the first hit. The new ray should now be able to hit the object moving in front of the second camera.

So two ray casts are necessary.

Some psuedo code that should work (some tweaks might be necessary, and this code can’t be copied into a source file and expect to compile)

ray = mainCamera.ScreenPointToRay(Input.mousePosition)
if (Physics.Raycast(ray, hit) && hit.object == theTVScreen)
{
   ray = secondCamera.ViewportPointToRay(new Vector3(hit.textureCoord));
   if (Physics.Raycast(ray, hit)
   {
      // hit should now contain information about what was hit through secondCamera
   }
}

Hi, @dorpeleg
Did you solve the the issue ? Could you please share your final code ? I have the same problem as you but cant make it work. I cant get a callback from the renderTexture at all with ray.

I know this is an old thread, but recently I need to do exact thing, raycasting from main camera to a ‘portal’ that has a render texture from a that ‘portal’ camera and want to share how I do this based on above clue:

void Update()
{
    RaycastHit hit;
    Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
  
    if (Physics.Raycast(ray, out hit))
    {
        Debug.DrawRay(ray.origin, ray.direction * 20, Color.green);
        if (hit.collider.tag == "portal")
        {
            Vector2 localPoint = hit.textureCoord;
            Ray _ray = cameraZoom.ViewportPointToRay(localPoint);
            RaycastHit _hit;
            if (Physics.Raycast(_ray, out _hit))
            {
                Debug.DrawRay(_ray.origin, _ray.direction * 20, Color.green);
                if (_hit.collider.tag == "dirt")
                {
                    print("contacted ");
                }
            }
        }
    }
}

I use an 3D object from my animation package and assign UV texture to it. Export it as FBX and import to Unity, then use that as my ‘portal’ view and assign Render Texture as its texture. The result is very precise.

If this is projected as a render texture to a raw image, use a RectTransformUtility.ScreenPointToLocalPointInRectangle() solution instead of having to do two raycasts

I had to do the something similar but for physical object in a BaseRaycaster, so the EventSystem can send the MouseOver and others … Hope it helps Component to pass Input event like PointerEnter to object displayed throught a RenderTexture · GitHub