Block raycast when UI is hit before

Questions & Answers
1

Hello,

I am using a Physics.Raycast() with terrain layer. But sometimes in game a Dialog appears with UI elements (and UI layer) and both UI buttons and raycasts work. I want to “turn off” the Raycasting, as this Dialog is shown.
Is there any nice way to do it?

What have I tried:

  1. Global static bool to know, when the Dialog is up - terrible, as dialog needs to set it true/false and if some other custom UI pops on camera, it needs to do the same (maybe semaphore then, no bool, but it’s still terrible).
  2. Check for UI plus Terrain layer masks in raycasts - terrible, generates boilerplate code and needs to join layer masks and then differentiate them again (and even doesn’t work, I don’t know, but it looks like the raycast hits terrain first using Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition))
  3. Don’t use layer masks at all - terrible, as Raycasting shall be adjusted to be used with layer masks.
  4. Running raycast function twice - as bad as it sounds, as it is called in Update.
  5. Graphic Raycaster on Dialog with ‘Blocking Objects’ sets to ‘All’ makes no difference in the outcome.
  6. Event system is telling me, I am clicking solely on my Dialog.

My original code is simply this:

if (Physics.Raycast(ray, out RaycastHit hit, Mathf.Infinity, terrainShiftedLayerMask))  
       {  
           return hit.point;  
       }

Code from my second point is this:

if (Physics.Raycast(ray, out RaycastHit hit, Mathf.Infinity, terrainAndUIShiftedLayerMask) &&  
       hit.collider.gameObject.layer != uiLayerMask) // If we hit UI, skip  
       {  
           return hit.point;  
       }
2

For everyone reading my question, as no answers are coming, I solved this by using my first mentioned solution (unfortunately).

After a quick research I have found out, you are not able to call raycast on one layer mask and expect it to hit another layer mask (as expected). And as I am raycasting more times on different layer masks (not only terrain), I have created a CustomRaycaster class with my Raycast function, that calls Physics.Raycast only when the raycasts are allowed. There is my snippet:

public class CustomRaycaster
{
    public static bool raycastingAllowed = true;

    public static bool Raycast(Ray ray, out RaycastHit hitInfo, float maxDistance, int layerMask)
    {
        if (!raycastingAllowed)
        {
            hitInfo = default;
            return false;
        }
        return Physics.Raycast(ray, out hitInfo, maxDistance, layerMask);
    }
}